2024-07-27T14:23:21.297 INFO:root:teuthology version: 0.0.1.dev361+g6e2a3ae 2024-07-27T14:23:21.297 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --verbose --name teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677 -- /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/orig.config.yaml 2024-07-27T14:23:21.337 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-27T14:23:21.423 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677 branch: squid description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: ceph-qa@ceph.com first_in_suite: false job_id: '7811677' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 2398 sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 sleep_before_teardown: 0 subset: 24756/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMN4uEnQYoPpP0A8dkQqVddzKoizfQbMSRV+vQm76Ua3NO0mr2a2BRH7I39sCOwy7G1pIkGDNDIKxvUFybPV0QE= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG/IfKsdn396O83xVDh4jpveUIu3IKKnftDh7EuLiTTfvVOFur/rptbxN+zbFKJ6gi5tCAuUDFv5zuBicf1eFk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e2a3aea75254d002e088a3ceee2a4fe2e292e3e timestamp: 2024-07-21_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-27T14:23:21.423 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa; will attempt to use it 2024-07-27T14:23:21.424 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks 2024-07-27T14:23:21.424 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-27T14:23:21.425 INFO:teuthology.task.internal:Checking packages... 2024-07-27T14:23:21.446 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '57fd7103c78964675e1941b32d58041fb7c15af7' 2024-07-27T14:23:21.446 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-27T14:23:21.446 INFO:teuthology.packaging:ref: None 2024-07-27T14:23:21.446 INFO:teuthology.packaging:tag: None 2024-07-27T14:23:21.446 INFO:teuthology.packaging:branch: squid 2024-07-27T14:23:21.446 INFO:teuthology.packaging:sha1: 57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:23:21.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-07-27T14:23:21.647 INFO:teuthology.task.internal:Found packages for ceph version 19.1.0-1142.gcc495590 2024-07-27T14:23:21.649 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-27T14:23:21.656 INFO:teuthology.task.internal:no buildpackages task found 2024-07-27T14:23:21.656 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-27T14:23:21.673 INFO:teuthology.task.internal:Saving configuration 2024-07-27T14:23:21.688 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-27T14:23:21.695 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-27T14:23:21.721 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-27 14:18:21.466160', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMN4uEnQYoPpP0A8dkQqVddzKoizfQbMSRV+vQm76Ua3NO0mr2a2BRH7I39sCOwy7G1pIkGDNDIKxvUFybPV0QE='} 2024-07-27T14:23:21.747 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-27 14:18:21.464781', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG/IfKsdn396O83xVDh4jpveUIu3IKKnftDh7EuLiTTfvVOFur/rptbxN+zbFKJ6gi5tCAuUDFv5zuBicf1eFk='} 2024-07-27T14:23:21.747 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-27T14:23:21.754 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-07-27T14:23:21.754 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-07-27T14:23:21.754 INFO:teuthology.run_tasks:Running task console_log... 2024-07-27T14:23:21.820 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcd5ef73ac0>, signals=[15]) 2024-07-27T14:23:21.820 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-27T14:23:21.827 INFO:teuthology.task.internal:Opening connections... 2024-07-27T14:23:21.827 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:23:21.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:23:21.916 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:23:21.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:23:22.041 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-27T14:23:22.048 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-07-27T14:23:22.066 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-07-27T14:23:22.067 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-07-27T14:23:22.123 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-07-27T14:23:22.124 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-27T14:23:22.125 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-27T14:23:22.125 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-27T14:23:22.125 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-07-27T14:23:22.148 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-07-27T14:23:22.166 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-07-27T14:23:22.166 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-07-27T14:23:22.223 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-27T14:23:22.224 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-27T14:23:22.224 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-07-27T14:23:22.248 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-27T14:23:22.257 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-27T14:23:22.262 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-27T14:23:22.262 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-07-27T14:23:22.264 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-07-27T14:23:22.281 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-27T14:23:22.287 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-27T14:23:22.287 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-07-27T14:23:22.322 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-07-27T14:23:22.561 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-27T14:23:22.616 INFO:teuthology.run_tasks:Running task kernel... 2024-07-27T14:23:22.632 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-07-27T14:23:22.632 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-07-27T14:23:22.633 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-27T14:23:22.633 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-27T14:23:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T14:23:22.650 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-07-27T14:23:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T14:23:22.651 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-07-27T14:23:22.707 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-437.el9.x86_64 2024-07-27T14:23:22.708 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-437.el9.x86_64 2024-07-27T14:23:22.708 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-07-27T14:23:22.709 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-437.el9.x86_64 2024-07-27T14:23:22.709 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.14.0-437.el9.x86_64 2024-07-27T14:23:22.709 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-07-27T14:23:24.949 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-07-27T14:23:25.231 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.2 MB 00:00 2024-07-27T14:23:27.392 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-07-27T14:23:28.190 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-07-27T14:23:32.932 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.5 MB 00:00 2024-07-27T14:23:33.975 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-07-27T14:23:34.566 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 18 kB 00:00 2024-07-27T14:23:35.604 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 24 MB/s | 22 MB 00:00 2024-07-27T14:23:35.784 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 18 kB 00:00 2024-07-27T14:23:37.188 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 17 MB/s | 22 MB 00:01 2024-07-27T14:23:41.109 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-07-27T14:23:42.519 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-27T14:23:43.091 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-27T14:23:43.091 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-27T14:23:43.190 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-27T14:23:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:23:43.196 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 84 M 2024-07-27T14:23:43.197 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 126 M 2024-07-27T14:23:43.197 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:23:43.645 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 4.5 MB/s | 1.5 MB 00:00 2024-07-27T14:23:44.428 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 15 MB/s | 17 MB 00:01 2024-07-27T14:23:44.542 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-27T14:23:44.543 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-27T14:23:44.636 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:23:44.645 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:23:44.645 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-27T14:23:44.646 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 84 M 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 126 M 2024-07-27T14:23:44.647 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:23:44.946 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-07-27T14:23:44.970 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 23 MB/s | 30 MB 00:01 2024-07-27T14:23:45.230 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 33 MB/s | 17 MB 00:00 2024-07-27T14:23:45.287 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 18 MB/s | 36 MB 00:01 2024-07-27T14:23:45.287 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:23:45.288 INFO:teuthology.orchestra.run.smithi112.stdout:Total 40 MB/s | 84 MB 00:02 2024-07-27T14:23:45.638 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 43 MB/s | 30 MB 00:00 2024-07-27T14:23:45.950 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 29 MB/s | 36 MB 00:01 2024-07-27T14:23:45.950 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:23:45.950 INFO:teuthology.orchestra.run.smithi087.stdout:Total 65 MB/s | 84 MB 00:01 2024-07-27T14:23:46.025 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:23:46.380 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:23:46.380 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:23:46.675 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:23:47.019 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:23:47.019 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:23:47.390 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:23:47.390 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:23:48.022 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:23:48.022 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:23:49.822 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:23:50.476 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-27T14:23:50.495 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:23:50.579 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:23:51.165 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-27T14:23:51.255 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:23:53.233 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:23:53.383 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:23:53.881 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:23:54.025 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:23:58.237 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:23:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:23:58.861 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:23:58.926 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:03.299 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:03.765 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.147 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.186 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.486 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.486 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-27T14:24:33.486 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:24:33.486 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:24:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.887 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:33.887 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-27T14:24:33.887 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T14:24:33.887 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.269 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:24:34.270 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:24:34.689 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:24:34.717 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:24:34.786 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.787 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T14:24:34.787 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-27T14:24:34.787 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-27T14:24:34.787 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-27T14:24:34.787 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-07-27T14:24:34.787 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-07-27T14:24:34.788 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T14:24:34.847 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:34.847 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T14:24:34.848 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-07-27T14:24:34.878 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-07-27T14:24:34.878 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.06 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 68.el9 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Group : Unspecified 2024-07-27T14:24:34.879 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 8264254 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : builder@centos.org 2024-07-27T14:24:34.880 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-27T14:24:34.881 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-27T14:24:34.882 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-27T14:24:34.882 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-07-27T14:24:34.882 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:24:34.882 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-07-27T14:24:34.884 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-27T14:24:34.884 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 5.14.0-480.el9.x86_64 2024-07-27T14:24:34.885 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-27T14:24:35.087 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T14:24:35.167 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:35.167 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T14:24:35.167 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-27T14:24:35.167 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-27T14:24:35.167 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-27T14:24:35.167 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-07-27T14:24:35.168 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-07-27T14:24:35.168 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T14:24:35.224 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T14:24:35.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T14:24:35.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-07-27T14:24:35.254 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-27T14:24:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:24:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-27T14:24:35.257 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-27T14:24:35.257 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-27T14:24:35.257 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-07-27T14:24:35.257 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:24:35.257 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-07-27T14:24:35.258 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-27T14:24:35.258 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-480.el9.x86_64 2024-07-27T14:24:35.259 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-27T14:24:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-07-27T14:24:35.875 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-07-27T14:24:36.943 INFO:teuthology.orchestra.run.smithi112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-27T14:24:36.983 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-07-27T14:24:36.985 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-07-27T14:24:37.000 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.r6OjjlLBH1 2024-07-27T14:24:37.001 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.r6OjjlLBH1 2024-07-27T14:24:37.067 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.r6OjjlLBH1 2024-07-27T14:24:37.199 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.r6OjjlLBH1 is 6KB 2024-07-27T14:24:37.248 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.r6OjjlLBH1 2024-07-27T14:24:37.249 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-27T14:24:37.263 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-07-27T14:24:37.269 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-07-27T14:24:37.271 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-07-27T14:24:37.289 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.fnhKgu5sOV 2024-07-27T14:24:37.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fnhKgu5sOV 2024-07-27T14:24:37.325 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-27T14:24:37.326 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-27T14:24:37.326 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-27T14:24:37.326 INFO:teuthology.orchestra.run.smithi112.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-27T14:24:37.328 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-27T14:24:37.353 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.fnhKgu5sOV 2024-07-27T14:24:37.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-07-27T14:24:37.492 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.fnhKgu5sOV is 6KB 2024-07-27T14:24:37.540 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.fnhKgu5sOV 2024-07-27T14:24:37.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-07-27T14:24:37.626 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-27T14:24:37.627 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-27T14:24:37.627 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-27T14:24:37.627 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-27T14:24:37.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-27T14:24:37.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-07-27T14:25:07.498 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-07-27T14:25:07.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:25:07.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:25:07.777 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-07-27T14:25:07.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:25:07.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:25:25.987 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-07-27T14:25:26.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-07-27T14:25:34.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:25:34.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:25:35.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:25:35.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:25:38.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-07-27T14:25:38.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-07-27T14:25:50.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:25:50.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:25:50.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:25:50.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:26:08.735 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-07-27T14:26:21.929 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-07-27T14:26:22.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:26:22.312 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-07-27T14:26:22.313 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-07-27T14:26:22.335 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-480.el9.x86_64 2024-07-27T14:26:22.335 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-27T14:26:22.335 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-27T14:26:22.335 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-27T14:26:23.336 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-07-27T14:26:23.336 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-27T14:26:23.403 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-07-27T14:26:23.446 DEBUG:teuthology.parallel:result is None 2024-07-27T14:26:23.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:26:23.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:26:24.121 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-07-27T14:26:24.590 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:26:24.590 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-07-27T14:26:24.590 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-07-27T14:26:24.605 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-480.el9.x86_64 2024-07-27T14:26:24.605 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-27T14:26:24.605 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-27T14:26:24.606 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-27T14:26:25.606 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-07-27T14:26:25.607 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-27T14:26:25.684 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-07-27T14:26:25.728 DEBUG:teuthology.parallel:result is None 2024-07-27T14:26:25.729 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-27T14:26:25.736 INFO:teuthology.task.internal:Creating test directory... 2024-07-27T14:26:25.737 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-27T14:26:25.739 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-27T14:26:25.787 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-27T14:26:25.794 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-27T14:26:25.872 INFO:teuthology.task.internal:Creating archive directory... 2024-07-27T14:26:25.872 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-27T14:26:25.876 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-27T14:26:25.953 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-27T14:26:25.959 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-27T14:26:25.959 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-27T14:26:25.962 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-27T14:26:26.030 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T14:26:26.052 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T14:26:26.109 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T14:26:26.120 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T14:26:26.121 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-27T14:26:26.129 INFO:teuthology.task.internal:Configuring sudo... 2024-07-27T14:26:26.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-27T14:26:26.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-27T14:26:26.194 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-27T14:26:26.203 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-27T14:26:26.204 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-27T14:26:26.206 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-27T14:26:26.241 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T14:26:26.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T14:26:26.394 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T14:26:26.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T14:26:26.475 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:26:26.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-27T14:26:26.538 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T14:26:26.607 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T14:26:26.641 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T14:26:26.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T14:26:26.758 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:26:26.758 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-27T14:26:26.820 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-07-27T14:26:26.823 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-07-27T14:26:26.866 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T14:26:26.922 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T14:26:27.177 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-27T14:26:27.185 INFO:teuthology.task.internal:Starting timer... 2024-07-27T14:26:27.185 INFO:teuthology.run_tasks:Running task pcp... 2024-07-27T14:26:27.263 INFO:teuthology.run_tasks:Running task selinux... 2024-07-27T14:26:27.399 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-07-27T14:26:27.400 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-07-27T14:26:27.464 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-07-27T14:26:27.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-07-27T14:26:27.522 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-07-27T14:26:27.523 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-27T14:26:27.523 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-07-27T14:26:27.543 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-07-27T14:26:27.544 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-07-27T14:26:27.565 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-07-27T14:26:27.565 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2024-07-27T14:26:27.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T14:26:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T14:26:27.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T14:26:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T14:26:27.633 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-27T14:26:27.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-07-27T14:26:27.674 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-07-27T14:26:27.701 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-27T14:26:27.711 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-27T14:26:27.722 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-27T14:26:27.723 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-27T14:30:59.118 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2024-07-27T14:30:59.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:30:59.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:30:59.199 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-07-27T14:30:59.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-07-27T14:30:59.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:30:59.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T14:30:59.352 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-07-27T14:30:59.430 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-27T14:30:59.431 INFO:teuthology.run_tasks:Running task clock... 2024-07-27T14:30:59.442 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-27T14:30:59.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-27T14:30:59.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T14:30:59.445 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-27T14:30:59.445 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T14:30:59.474 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-27T14:30:59.487 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-27T14:30:59.500 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-27T14:30:59.510 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-07-27T14:30:59.514 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-27T14:30:59.521 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-07-27T14:30:59.535 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-27T14:30:59.538 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-07-27T14:30:59.549 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-27T14:30:59.549 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-07-27T14:30:59.563 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-27T14:30:59.578 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-27T14:30:59.615 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-07-27T14:30:59.619 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T14:30:59.620 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-07-27T14:30:59.620 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.620 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.620 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.620 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.638 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-07-27T14:30:59.639 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T14:30:59.639 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-07-27T14:30:59.639 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.639 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.639 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.640 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T14:30:59.641 INFO:teuthology.run_tasks:Running task install... 2024-07-27T14:30:59.651 DEBUG:teuthology.task.install:project ceph 2024-07-27T14:30:59.651 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7'}} 2024-07-27T14:30:59.651 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7'} 2024-07-27T14:30:59.651 INFO:teuthology.task.install:Using flavor: default 2024-07-27T14:30:59.659 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-07-27T14:30:59.659 INFO:teuthology.task.install:extra packages: [] 2024-07-27T14:30:59.659 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7', 'tag': None, 'wait_for_package': False} 2024-07-27T14:30:59.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:30:59.662 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7', 'tag': None, 'wait_for_package': False} 2024-07-27T14:30:59.662 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:30:59.813 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/ 2024-07-27T14:30:59.814 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3573.g57fd7103 2024-07-27T14:30:59.817 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/ 2024-07-27T14:30:59.817 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3573.g57fd7103 2024-07-27T14:30:59.928 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-27T14:30:59.928 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:30:59.928 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-27T14:30:59.937 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/57fd7103c78964675e1941b32d58041fb7c15af7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-27T14:30:59.938 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:30:59.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-27T14:30:59.955 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-07-27T14:30:59.956 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/57fd7103c78964675e1941b32d58041fb7c15af7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-27T14:30:59.966 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-07-27T14:30:59.966 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/57fd7103c78964675e1941b32d58041fb7c15af7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-27T14:31:00.022 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-27T14:31:00.037 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-27T14:31:00.096 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-27T14:31:00.111 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-27T14:31:00.160 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-07-27T14:31:00.162 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-07-27T14:31:00.175 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-07-27T14:31:00.177 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-07-27T14:31:00.406 INFO:teuthology.orchestra.run.smithi087.stdout:45 files removed 2024-07-27T14:31:00.414 INFO:teuthology.orchestra.run.smithi112.stdout:45 files removed 2024-07-27T14:31:00.433 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-07-27T14:31:00.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2024-07-27T14:31:01.040 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 257 kB/s | 83 kB 00:00 2024-07-27T14:31:01.050 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 251 kB/s | 83 kB 00:00 2024-07-27T14:31:01.383 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-07-27T14:31:01.390 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 39 kB/s | 12 kB 00:00 2024-07-27T14:31:01.699 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 7.2 kB/s | 1.9 kB 00:00 2024-07-27T14:31:01.717 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-07-27T14:31:02.218 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-07-27T14:31:03.274 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.2 MB 00:01 2024-07-27T14:31:06.286 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-07-27T14:31:06.852 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-07-27T14:31:11.487 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-07-27T14:31:11.777 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.5 MB 00:01 2024-07-27T14:31:12.968 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 18 kB 00:00 2024-07-27T14:31:13.586 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 18 kB 00:00 2024-07-27T14:31:14.147 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-07-27T14:31:14.949 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 18 MB/s | 22 MB 00:01 2024-07-27T14:31:19.545 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-07-27T14:31:20.297 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-07-27T14:31:21.896 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:31:21.910 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:21.910 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:31:21.910 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:21.910 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 10 M 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.4 M 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.2 M 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 21 M 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 25 k 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-27T14:31:21.911 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 751 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 506 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 45 k 2024-07-27T14:31:21.912 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 136 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 162 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 320 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 301 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 99 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:21.913 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 60 M 2024-07-27T14:31:21.914 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:31:22.643 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 10 M 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-07-27T14:31:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.4 M 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.2 M 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 21 M 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 25 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 751 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-27T14:31:22.664 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 506 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 45 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 136 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 162 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-27T14:31:22.665 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 320 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 301 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 99 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout:Install 30 Packages 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-07-27T14:31:22.666 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:22.667 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 60 M 2024-07-27T14:31:22.667 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:31:22.946 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-base-19.0.0-3573.g57fd7103.el9.x86 9.6 MB/s | 5.4 MB 00:00 2024-07-27T14:31:23.030 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.0.0-3573.g57fd7103.el9. 301 kB/s | 25 kB 00:00 2024-07-27T14:31:23.122 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.0.0-3573.g57fd7103.el9.x8 8.0 MB/s | 751 kB 00:00 2024-07-27T14:31:23.239 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-radosgw-19.0.0-3573.g57fd7103.el9. 12 MB/s | 10 MB 00:00 2024-07-27T14:31:23.264 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): libradosstriper1-19.0.0-3573.g57fd7103. 3.5 MB/s | 506 kB 00:00 2024-07-27T14:31:23.323 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.0.0-3573.g57fd 774 kB/s | 45 kB 00:00 2024-07-27T14:31:23.406 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.0.0-3573.g57fd71 1.6 MB/s | 136 kB 00:00 2024-07-27T14:31:23.507 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): librgw2-19.0.0-3573.g57fd7103.el9.x86_6 20 MB/s | 5.4 MB 00:00 2024-07-27T14:31:23.532 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-cephfs-19.0.0-3573.g57fd7103.el 1.3 MB/s | 162 kB 00:00 2024-07-27T14:31:23.557 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-rados-19.0.0-3573.g57fd7103.el 6.2 MB/s | 320 kB 00:00 2024-07-27T14:31:23.582 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rbd-19.0.0-3573.g57fd7103.el9. 5.9 MB/s | 301 kB 00:00 2024-07-27T14:31:23.608 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rgw-19.0.0-3573.g57fd7103.el9. 1.9 MB/s | 99 kB 00:00 2024-07-27T14:31:23.825 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): ceph-common-19.0.0-3573.g57fd7103.el9. 15 MB/s | 21 MB 00:01 2024-07-27T14:31:24.048 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-base-19.0.0-3573.g57fd7103.el9.x86 4.3 MB/s | 5.4 MB 00:01 2024-07-27T14:31:24.090 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-selinux-19.0.0-3573.g57fd7103.el9. 608 kB/s | 25 kB 00:00 2024-07-27T14:31:24.167 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 183 kB/s | 107 kB 00:00 2024-07-27T14:31:24.190 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): libcephfs2-19.0.0-3573.g57fd7103.el9.x8 7.3 MB/s | 751 kB 00:00 2024-07-27T14:31:24.192 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 123 kB/s | 45 kB 00:00 2024-07-27T14:31:24.250 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 249 kB/s | 160 kB 00:00 2024-07-27T14:31:24.265 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): libradosstriper1-19.0.0-3573.g57fd7103. 6.6 MB/s | 506 kB 00:00 2024-07-27T14:31:24.368 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-07-27T14:31:24.457 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-radosgw-19.0.0-3573.g57fd7103.el9. 6.3 MB/s | 10 MB 00:01 2024-07-27T14:31:24.508 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): python3-ceph-argparse-19.0.0-3573.g57fd 906 kB/s | 45 kB 00:00 2024-07-27T14:31:24.526 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-07-27T14:31:24.551 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 814 kB/s | 292 kB 00:00 2024-07-27T14:31:24.566 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-ceph-common-19.0.0-3573.g57fd71 2.3 MB/s | 136 kB 00:00 2024-07-27T14:31:24.577 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 221 kB 00:00 2024-07-27T14:31:24.625 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-cephfs-19.0.0-3573.g57fd7103.el 2.7 MB/s | 162 kB 00:00 2024-07-27T14:31:24.635 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 330 kB/s | 36 kB 00:00 2024-07-27T14:31:24.661 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 435 kB/s | 47 kB 00:00 2024-07-27T14:31:24.725 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): librgw2-19.0.0-3573.g57fd7103.el9.x86_6 12 MB/s | 5.4 MB 00:00 2024-07-27T14:31:24.750 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-rados-19.0.0-3573.g57fd7103.el 2.5 MB/s | 320 kB 00:00 2024-07-27T14:31:24.752 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-07-27T14:31:24.778 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-07-27T14:31:24.784 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-rbd-19.0.0-3573.g57fd7103.el9. 5.0 MB/s | 301 kB 00:00 2024-07-27T14:31:24.803 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-07-27T14:31:24.809 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-rgw-19.0.0-3573.g57fd7103.el9. 1.7 MB/s | 99 kB 00:00 2024-07-27T14:31:24.828 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-07-27T14:31:24.870 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-07-27T14:31:24.912 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-07-27T14:31:24.987 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-07-27T14:31:25.010 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): ceph-common-19.0.0-3573.g57fd7103.el9. 9.6 MB/s | 21 MB 00:02 2024-07-27T14:31:25.113 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-07-27T14:31:25.180 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): librados2-19.0.0-3573.g57fd7103.el9.x8 13 MB/s | 3.4 MB 00:00 2024-07-27T14:31:25.297 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.0.0-3573.g57fd7103.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-07-27T14:31:25.301 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:31:25.301 INFO:teuthology.orchestra.run.smithi087.stdout:Total 18 MB/s | 60 MB 00:03 2024-07-27T14:31:25.399 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:31:25.434 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:31:25.434 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:31:25.460 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 158 kB/s | 107 kB 00:00 2024-07-27T14:31:25.493 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 104 kB/s | 45 kB 00:00 2024-07-27T14:31:25.535 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2024-07-27T14:31:25.794 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 163 kB/s | 42 kB 00:00 2024-07-27T14:31:25.819 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-07-27T14:31:25.844 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 834 kB/s | 292 kB 00:00 2024-07-27T14:31:25.903 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-07-27T14:31:25.928 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 330 kB/s | 36 kB 00:00 2024-07-27T14:31:25.954 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 435 kB/s | 47 kB 00:00 2024-07-27T14:31:26.000 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:31:26.000 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:31:26.079 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-07-27T14:31:26.104 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 257 kB/s | 26 kB 00:00 2024-07-27T14:31:26.129 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-07-27T14:31:26.155 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-07-27T14:31:26.188 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-07-27T14:31:26.214 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-07-27T14:31:26.306 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-27T14:31:26.390 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-07-27T14:31:26.540 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): librados2-19.0.0-3573.g57fd7103.el9.x8 10 MB/s | 3.4 MB 00:00 2024-07-27T14:31:26.590 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): librbd1-19.0.0-3573.g57fd7103.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-07-27T14:31:26.596 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:31:26.596 INFO:teuthology.orchestra.run.smithi112.stdout:Total 15 MB/s | 60 MB 00:03 2024-07-27T14:31:26.691 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:31:26.727 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:31:26.727 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:31:27.299 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:31:27.299 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:31:27.423 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:31:27.493 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-07-27T14:31:27.527 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-07-27T14:31:27.560 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-07-27T14:31:27.583 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-27T14:31:27.607 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-27T14:31:27.760 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-27T14:31:27.813 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-27T14:31:28.207 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-07-27T14:31:28.262 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:28.355 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:28.445 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 8/34 2024-07-27T14:31:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 9/34 2024-07-27T14:31:28.535 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 9/34 2024-07-27T14:31:28.564 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-07-27T14:31:28.608 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9. 11/34 2024-07-27T14:31:28.678 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 12/34 2024-07-27T14:31:28.699 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 13/34 2024-07-27T14:31:28.714 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:31:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 13/34 2024-07-27T14:31:28.806 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-07-27T14:31:28.823 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-07-27T14:31:28.845 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-07-27T14:31:28.870 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-07-27T14:31:28.893 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-27T14:31:28.920 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-27T14:31:28.934 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-07-27T14:31:28.966 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-07-27T14:31:29.007 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-07-27T14:31:29.039 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-07-27T14:31:29.068 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-07-27T14:31:29.070 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-27T14:31:29.101 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-07-27T14:31:29.123 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-27T14:31:29.423 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-07-27T14:31:29.472 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 22/34 2024-07-27T14:31:29.489 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-07-27T14:31:29.563 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 22/34 2024-07-27T14:31:29.572 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:29.594 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 23/34 2024-07-27T14:31:29.630 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-07-27T14:31:29.665 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:29.754 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 8/34 2024-07-27T14:31:29.782 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 9/34 2024-07-27T14:31:29.845 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 9/34 2024-07-27T14:31:29.874 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-07-27T14:31:29.918 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9. 11/34 2024-07-27T14:31:29.987 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 12/34 2024-07-27T14:31:30.009 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 13/34 2024-07-27T14:31:30.057 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 13/34 2024-07-27T14:31:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-07-27T14:31:30.110 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-07-27T14:31:30.133 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 26/34 2024-07-27T14:31:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-07-27T14:31:30.203 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 26/34 2024-07-27T14:31:30.251 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-07-27T14:31:30.274 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 27/34 2024-07-27T14:31:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-07-27T14:31:30.302 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x8 28/34 2024-07-27T14:31:30.326 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-07-27T14:31:30.354 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-07-27T14:31:30.387 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-07-27T14:31:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-07-27T14:31:30.733 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 22/34 2024-07-27T14:31:30.824 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 22/34 2024-07-27T14:31:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 23/34 2024-07-27T14:31:30.890 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-07-27T14:31:31.363 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-07-27T14:31:31.394 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 26/34 2024-07-27T14:31:31.448 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 26/34 2024-07-27T14:31:31.518 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 27/34 2024-07-27T14:31:31.546 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x8 28/34 2024-07-27T14:31:32.606 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:32.641 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:33.220 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:33.253 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 30/34 2024-07-27T14:31:33.334 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 30/34 2024-07-27T14:31:33.334 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-27T14:31:33.334 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-27T14:31:33.334 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:33.363 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:33.826 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:33.944 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:34.510 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 29/34 2024-07-27T14:31:34.556 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 30/34 2024-07-27T14:31:34.646 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 30/34 2024-07-27T14:31:34.647 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-27T14:31:34.647 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-27T14:31:34.647 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:34.673 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:42.977 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-07-27T14:31:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:43.960 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 32/34 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 32/34 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T14:31:43.985 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:43.987 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-27T14:31:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-27T14:31:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:44.384 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-07-27T14:31:44.384 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-07-27T14:31:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:45.240 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 1/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 3/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 4/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 5/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 6/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9. 8/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x8 9/34 2024-07-27T14:31:45.241 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 10/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 11/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 12/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 13/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-07-27T14:31:45.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-07-27T14:31:45.243 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-07-27T14:31:45.243 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-07-27T14:31:45.243 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-07-27T14:31:45.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 33/34 2024-07-27T14:31:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 32/34 2024-07-27T14:31:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 32/34 2024-07-27T14:31:45.392 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:31:45.392 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-27T14:31:45.392 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T14:31:45.392 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T14:31:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-27T14:31:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-27T14:31:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:31:46.146 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.147 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-27T14:31:46.148 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-27T14:31:46.149 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2024-07-27T14:31:46.149 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-27T14:31:46.149 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:46.149 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:31:46.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 1/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 3/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 4/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 5/34 2024-07-27T14:31:46.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_6 6/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 7/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9. 8/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x8 9/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 10/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 11/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 12/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 13/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-07-27T14:31:46.628 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-07-27T14:31:46.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 31/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-07-27T14:31:46.630 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 33/34 2024-07-27T14:31:47.070 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:28 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:31:47.196 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:31:47.196 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:47.196 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:31:47.196 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:47.196 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 49 M 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 163 k 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-07-27T14:31:47.197 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:47.198 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 49 M 2024-07-27T14:31:47.198 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 208 M 2024-07-27T14:31:47.198 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:31:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-27T14:31:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:47.517 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:31:47.575 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): libcephsqlite-19.0.0-3573.g57fd7103.el9. 639 kB/s | 163 kB 00:00 2024-07-27T14:31:47.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2024-07-27T14:31:48.225 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 336 kB/s | 304 kB 00:00 2024-07-27T14:31:48.438 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:28 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:31:48.475 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 73 kB/s | 64 kB 00:00 2024-07-27T14:31:48.563 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 49 M 2024-07-27T14:31:48.564 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 163 k 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-07-27T14:31:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:48.566 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 49 M 2024-07-27T14:31:48.566 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 208 M 2024-07-27T14:31:48.566 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:31:48.928 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-07-27T14:31:48.953 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): libcephsqlite-19.0.0-3573.g57fd7103.el9. 606 kB/s | 163 kB 00:00 2024-07-27T14:31:48.970 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-07-27T14:31:49.092 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): ceph-test-19.0.0-3573.g57fd7103.el9.x86_ 27 MB/s | 49 MB 00:01 2024-07-27T14:31:49.094 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:31:49.095 INFO:teuthology.orchestra.run.smithi087.stdout:Total 26 MB/s | 49 MB 00:01 2024-07-27T14:31:49.122 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:31:49.142 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:31:49.142 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:31:49.510 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:31:49.511 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:31:50.043 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:31:50.119 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-07-27T14:31:50.157 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-07-27T14:31:50.186 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 3/4 2024-07-27T14:31:50.337 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): ceph-test-19.0.0-3573.g57fd7103.el9.x86_ 29 MB/s | 49 MB 00:01 2024-07-27T14:31:50.339 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:31:50.340 INFO:teuthology.orchestra.run.smithi112.stdout:Total 28 MB/s | 49 MB 00:01 2024-07-27T14:31:50.367 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:31:50.387 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:31:50.387 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:31:50.755 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:31:50.755 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:31:51.279 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:31:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-07-27T14:31:51.411 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-07-27T14:31:51.439 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 3/4 2024-07-27T14:31:54.771 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 3/4 2024-07-27T14:31:54.860 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 4/4 2024-07-27T14:31:56.036 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 3/4 2024-07-27T14:31:56.113 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 4/4 2024-07-27T14:31:57.238 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 4/4 2024-07-27T14:31:57.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 1/4 2024-07-27T14:31:57.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 2/4 2024-07-27T14:31:57.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-07-27T14:31:57.498 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-07-27T14:31:57.498 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:57.499 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:31:57.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-07-27T14:31:58.374 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:39 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:31:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 4/4 2024-07-27T14:31:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 1/4 2024-07-27T14:31:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 2/4 2024-07-27T14:31:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-07-27T14:31:58.515 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:31:58.517 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:58.517 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:31:58.517 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:58.517 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 6.6 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 2.4 M 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.5 M 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 249 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 4.7 M 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 17 M 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-27T14:31:58.518 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-27T14:31:58.519 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:58.520 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout:Install 36 Packages 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 30 M 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 108 M 2024-07-27T14:31:58.521 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:31:58.642 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-07-27T14:31:58.642 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:58.643 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:31:58.897 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2024-07-27T14:31:59.344 INFO:teuthology.orchestra.run.smithi087.stdout:(1/36): ceph-19.0.0-3573.g57fd7103.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-07-27T14:31:59.464 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:39 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:31:59.544 INFO:teuthology.orchestra.run.smithi087.stdout:(2/36): ceph-mds-19.0.0-3573.g57fd7103.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-07-27T14:31:59.586 INFO:teuthology.orchestra.run.smithi087.stdout:(3/36): ceph-mgr-19.0.0-3573.g57fd7103.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-07-27T14:31:59.604 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 6.6 k 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 2.4 M 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.5 M 2024-07-27T14:31:59.607 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 249 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 4.7 M 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 17 M 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections 2024-07-27T14:31:59.608 INFO:teuthology.orchestra.run.smithi112.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-27T14:31:59.609 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout:Install 36 Packages 2024-07-27T14:31:59.610 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:31:59.611 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 M 2024-07-27T14:31:59.611 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 108 M 2024-07-27T14:31:59.611 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:31:59.662 INFO:teuthology.orchestra.run.smithi087.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3573.g57fd 3.2 MB/s | 249 kB 00:00 2024-07-27T14:31:59.828 INFO:teuthology.orchestra.run.smithi087.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-07-27T14:31:59.946 INFO:teuthology.orchestra.run.smithi087.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-07-27T14:32:00.104 INFO:teuthology.orchestra.run.smithi087.stdout:(7/36): ceph-osd-19.0.0-3573.g57fd7103.el9.x86_ 30 MB/s | 17 MB 00:00 2024-07-27T14:32:00.129 INFO:teuthology.orchestra.run.smithi087.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 581 kB/s | 106 kB 00:00 2024-07-27T14:32:00.204 INFO:teuthology.orchestra.run.smithi087.stdout:(9/36): ceph-mon-19.0.0-3573.g57fd7103.el9.x86_ 5.5 MB/s | 4.7 MB 00:00 2024-07-27T14:32:00.230 INFO:teuthology.orchestra.run.smithi087.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-07-27T14:32:00.289 INFO:teuthology.orchestra.run.smithi087.stdout:(11/36): python3-pycparser-2.20-6.el9.noarch.rp 734 kB/s | 135 kB 00:00 2024-07-27T14:32:00.373 INFO:teuthology.orchestra.run.smithi087.stdout:(12/36): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-07-27T14:32:00.464 INFO:teuthology.orchestra.run.smithi087.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 805 kB/s | 188 kB 00:00 2024-07-27T14:32:00.489 INFO:teuthology.orchestra.run.smithi087.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 297 kB/s | 35 kB 00:00 2024-07-27T14:32:00.515 INFO:teuthology.orchestra.run.smithi087.stdout:(15/36): python3-mako-1.1.4-6.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-07-27T14:32:00.540 INFO:teuthology.orchestra.run.smithi087.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-07-27T14:32:00.565 INFO:teuthology.orchestra.run.smithi087.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-07-27T14:32:00.641 INFO:teuthology.orchestra.run.smithi087.stdout:(18/36): python3-toml-0.10.2-6.el9.noarch.rpm 276 kB/s | 42 kB 00:00 2024-07-27T14:32:00.691 INFO:teuthology.orchestra.run.smithi087.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 347 kB/s | 43 kB 00:00 2024-07-27T14:32:00.716 INFO:teuthology.orchestra.run.smithi087.stdout:(20/36): luarocks-3.9.2-5.el9.noarch.rpm 861 kB/s | 151 kB 00:00 2024-07-27T14:32:00.741 INFO:teuthology.orchestra.run.smithi087.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-07-27T14:32:00.766 INFO:teuthology.orchestra.run.smithi087.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-07-27T14:32:00.792 INFO:teuthology.orchestra.run.smithi087.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-07-27T14:32:00.807 INFO:teuthology.orchestra.run.smithi112.stdout:(1/36): ceph-19.0.0-3573.g57fd7103.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-07-27T14:32:00.817 INFO:teuthology.orchestra.run.smithi087.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-07-27T14:32:00.842 INFO:teuthology.orchestra.run.smithi087.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-07-27T14:32:00.868 INFO:teuthology.orchestra.run.smithi087.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-07-27T14:32:00.893 INFO:teuthology.orchestra.run.smithi087.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-07-27T14:32:00.918 INFO:teuthology.orchestra.run.smithi087.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-07-27T14:32:00.943 INFO:teuthology.orchestra.run.smithi087.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-07-27T14:32:00.969 INFO:teuthology.orchestra.run.smithi087.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-07-27T14:32:00.973 INFO:teuthology.orchestra.run.smithi112.stdout:(2/36): ceph-mgr-19.0.0-3573.g57fd7103.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-07-27T14:32:00.994 INFO:teuthology.orchestra.run.smithi087.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-07-27T14:32:01.015 INFO:teuthology.orchestra.run.smithi112.stdout:(3/36): ceph-mds-19.0.0-3573.g57fd7103.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-07-27T14:32:01.019 INFO:teuthology.orchestra.run.smithi087.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-07-27T14:32:01.045 INFO:teuthology.orchestra.run.smithi087.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-07-27T14:32:01.070 INFO:teuthology.orchestra.run.smithi087.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-07-27T14:32:01.087 INFO:teuthology.orchestra.run.smithi087.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2024-07-27T14:32:01.090 INFO:teuthology.orchestra.run.smithi112.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3573.g57fd 3.3 MB/s | 249 kB 00:00 2024-07-27T14:32:01.120 INFO:teuthology.orchestra.run.smithi087.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-07-27T14:32:01.125 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:01.125 INFO:teuthology.orchestra.run.smithi087.stdout:Total 12 MB/s | 30 MB 00:02 2024-07-27T14:32:01.166 INFO:teuthology.orchestra.run.smithi112.stdout:(5/36): ceph-mon-19.0.0-3573.g57fd7103.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-07-27T14:32:01.260 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:01.300 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:01.301 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:01.541 INFO:teuthology.orchestra.run.smithi112.stdout:(6/36): ceph-osd-19.0.0-3573.g57fd7103.el9.x86_ 30 MB/s | 17 MB 00:00 2024-07-27T14:32:01.633 INFO:teuthology.orchestra.run.smithi112.stdout:(7/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 467 kB/s | 253 kB 00:00 2024-07-27T14:32:01.730 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:01.731 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:01.742 INFO:teuthology.orchestra.run.smithi112.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-07-27T14:32:01.851 INFO:teuthology.orchestra.run.smithi112.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-07-27T14:32:01.976 INFO:teuthology.orchestra.run.smithi112.stdout:(10/36): python3-urllib3-1.26.5-6.el9.noarch.rp 1.7 MB/s | 215 kB 00:00 2024-07-27T14:32:02.001 INFO:teuthology.orchestra.run.smithi112.stdout:(11/36): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-07-27T14:32:02.135 INFO:teuthology.orchestra.run.smithi112.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-07-27T14:32:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-07-27T14:32:02.935 INFO:teuthology.orchestra.run.smithi112.stdout:(14/36): lua-5.4.4-4.el9.x86_64.rpm 196 kB/s | 188 kB 00:00 2024-07-27T14:32:02.985 INFO:teuthology.orchestra.run.smithi112.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 41 kB/s | 35 kB 00:00 2024-07-27T14:32:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-07-27T14:32:03.085 INFO:teuthology.orchestra.run.smithi112.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-07-27T14:32:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-07-27T14:32:03.142 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-07-27T14:32:03.169 INFO:teuthology.orchestra.run.smithi112.stdout:(17/36): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-07-27T14:32:03.186 INFO:teuthology.orchestra.run.smithi112.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.6 MB/s | 43 kB 00:00 2024-07-27T14:32:03.211 INFO:teuthology.orchestra.run.smithi112.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-07-27T14:32:03.214 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-07-27T14:32:03.250 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-07-27T14:32:03.253 INFO:teuthology.orchestra.run.smithi112.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 8.4 MB/s | 358 kB 00:00 2024-07-27T14:32:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-07-27T14:32:03.295 INFO:teuthology.orchestra.run.smithi112.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 693 kB/s | 11 kB 00:00 2024-07-27T14:32:03.312 INFO:teuthology.orchestra.run.smithi112.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-07-27T14:32:03.329 INFO:teuthology.orchestra.run.smithi112.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 1.3 MB/s | 23 kB 00:00 2024-07-27T14:32:03.346 INFO:teuthology.orchestra.run.smithi112.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-07-27T14:32:03.362 INFO:teuthology.orchestra.run.smithi112.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 1.2 MB/s | 20 kB 00:00 2024-07-27T14:32:03.375 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-07-27T14:32:03.388 INFO:teuthology.orchestra.run.smithi112.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-07-27T14:32:03.413 INFO:teuthology.orchestra.run.smithi112.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2024-07-27T14:32:03.439 INFO:teuthology.orchestra.run.smithi112.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2024-07-27T14:32:03.464 INFO:teuthology.orchestra.run.smithi112.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-07-27T14:32:03.469 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-07-27T14:32:03.489 INFO:teuthology.orchestra.run.smithi112.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-07-27T14:32:03.510 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-07-27T14:32:03.515 INFO:teuthology.orchestra.run.smithi112.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-07-27T14:32:03.548 INFO:teuthology.orchestra.run.smithi112.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-07-27T14:32:03.553 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-07-27T14:32:03.582 INFO:teuthology.orchestra.run.smithi112.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 13 MB/s | 427 kB 00:00 2024-07-27T14:32:03.583 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-07-27T14:32:03.607 INFO:teuthology.orchestra.run.smithi112.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-07-27T14:32:03.618 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-07-27T14:32:03.651 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-07-27T14:32:03.691 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-07-27T14:32:03.707 INFO:teuthology.orchestra.run.smithi112.stdout:(35/36): python3-pytz-2021.1-5.el9.noarch.rpm 58 kB/s | 51 kB 00:00 2024-07-27T14:32:03.727 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-07-27T14:32:03.761 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-07-27T14:32:03.800 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-07-27T14:32:03.833 INFO:teuthology.orchestra.run.smithi112.stdout:(36/36): python3-toml-0.10.2-6.el9.noarch.rpm 46 kB/s | 42 kB 00:00 2024-07-27T14:32:03.862 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:03.862 INFO:teuthology.orchestra.run.smithi112.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-07-27T14:32:03.872 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-07-27T14:32:03.980 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:04.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-07-27T14:32:04.020 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:04.020 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:04.091 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-07-27T14:32:04.155 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 21/36 2024-07-27T14:32:04.191 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-07-27T14:32:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-07-27T14:32:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-07-27T14:32:04.480 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:04.481 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:04.501 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-07-27T14:32:04.553 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-07-27T14:32:04.632 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-07-27T14:32:04.720 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-07-27T14:32:04.768 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-07-27T14:32:04.887 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-07-27T14:32:05.049 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9. 31/36 2024-07-27T14:32:05.079 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 32/36 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 32/36 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T14:32:05.107 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:05.768 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-07-27T14:32:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-07-27T14:32:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-07-27T14:32:05.958 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-07-27T14:32:06.004 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-07-27T14:32:06.036 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-07-27T14:32:06.128 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-07-27T14:32:06.206 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-07-27T14:32:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-07-27T14:32:06.282 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-07-27T14:32:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-07-27T14:32:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-07-27T14:32:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-07-27T14:32:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-07-27T14:32:06.472 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-07-27T14:32:06.507 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-07-27T14:32:06.546 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-07-27T14:32:06.555 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 33/36 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 33/36 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T14:32:06.583 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:06.626 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-07-27T14:32:06.764 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-07-27T14:32:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-07-27T14:32:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 21/36 2024-07-27T14:32:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-07-27T14:32:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-07-27T14:32:07.031 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 34/36 2024-07-27T14:32:07.049 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 34/36 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T14:32:07.055 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:07.235 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-07-27T14:32:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-07-27T14:32:07.290 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 35/36 2024-07-27T14:32:07.314 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 35/36 2024-07-27T14:32:07.314 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:07.314 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-27T14:32:07.314 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T14:32:07.314 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T14:32:07.315 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:07.360 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-07-27T14:32:07.379 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 36/36 2024-07-27T14:32:07.448 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-07-27T14:32:07.494 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-07-27T14:32:07.614 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-07-27T14:32:07.778 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9. 31/36 2024-07-27T14:32:07.808 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 32/36 2024-07-27T14:32:07.838 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 32/36 2024-07-27T14:32:07.839 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:07.839 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-27T14:32:07.839 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T14:32:07.839 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T14:32:07.839 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:09.097 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 36/36 2024-07-27T14:32:09.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 1/36 2024-07-27T14:32:09.099 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 2/36 2024-07-27T14:32:09.099 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 3/36 2024-07-27T14:32:09.100 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 4/36 2024-07-27T14:32:09.100 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 5/36 2024-07-27T14:32:09.100 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9. 6/36 2024-07-27T14:32:09.100 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-07-27T14:32:09.100 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-07-27T14:32:09.101 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-07-27T14:32:09.102 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-07-27T14:32:09.103 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-07-27T14:32:09.104 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-07-27T14:32:09.105 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-07-27T14:32:09.105 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-07-27T14:32:09.105 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-07-27T14:32:09.270 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 33/36 2024-07-27T14:32:09.294 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 33/36 2024-07-27T14:32:09.295 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:09.295 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-27T14:32:09.295 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T14:32:09.295 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T14:32:09.295 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:09.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 34/36 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 34/36 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T14:32:09.759 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:09.994 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 35/36 2024-07-27T14:32:10.017 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 35/36 2024-07-27T14:32:10.018 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:10.018 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-27T14:32:10.018 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T14:32:10.018 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T14:32:10.018 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:10.086 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 36/36 2024-07-27T14:32:10.170 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-07-27T14:32:10.170 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-27T14:32:10.171 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-27T14:32:10.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-27T14:32:10.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:10.174 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:10.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-07-27T14:32:11.035 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:52 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:11.119 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:11.160 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:11.161 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:11.161 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:11.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-07-27T14:32:11.753 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:52 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 36/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 1/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 2/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 3/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 4/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 5/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9. 6/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-07-27T14:32:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-07-27T14:32:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-07-27T14:32:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-07-27T14:32:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-07-27T14:32:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-07-27T14:32:11.880 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 764 k 2024-07-27T14:32:11.881 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 764 k 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 770 k 2024-07-27T14:32:11.882 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:12.157 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.0.0-3573.g57fd7103.el9.noarch.rpm 2.7 MB/s | 764 kB 00:00 2024-07-27T14:32:12.157 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:12.157 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.7 MB/s | 764 kB 00:00 2024-07-27T14:32:12.158 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:12.162 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:12.163 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:12.168 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:12.169 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:12.241 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:12.676 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-27T14:32:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-27T14:32:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-27T14:32:12.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-27T14:32:12.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-27T14:32:12.733 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:12.733 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:12.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:13.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2024-07-27T14:32:13.168 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:13.448 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:13.449 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:13.449 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:13.449 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:13.449 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:13.449 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:13.654 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:53 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:13.660 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-07-27T14:32:13.739 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:13.781 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:13.782 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:32:13.782 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:13.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2024-07-27T14:32:14.215 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:55 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:14.344 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 145 k 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:14.345 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:14.346 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:32:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:14.346 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 145 k 2024-07-27T14:32:14.346 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 435 k 2024-07-27T14:32:14.346 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:14.383 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:54 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:14.509 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 764 k 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:14.510 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:32:14.511 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:14.511 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 764 k 2024-07-27T14:32:14.511 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 770 k 2024-07-27T14:32:14.511 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:14.555 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.0.0-3573.g57fd71 696 kB/s | 145 kB 00:00 2024-07-27T14:32:14.555 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:14.555 INFO:teuthology.orchestra.run.smithi087.stdout:Total 692 kB/s | 145 kB 00:00 2024-07-27T14:32:14.556 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:14.563 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:14.563 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:14.620 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:14.620 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:14.769 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:14.793 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-19.0.0-3573.g57fd7103.el9.noarch.rpm 2.6 MB/s | 764 kB 00:00 2024-07-27T14:32:14.794 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:14.794 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.6 MB/s | 764 kB 00:00 2024-07-27T14:32:14.794 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:14.800 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:14.800 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:14.806 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:14.806 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:14.823 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:14.848 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:14.848 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:14.848 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-27T14:32:14.848 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:14.877 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:15.346 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:15.447 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:15.773 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:15.773 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:15.773 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:15.773 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:15.773 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:15.774 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:15.802 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:16.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-07-27T14:32:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/1 2024-07-27T14:32:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:16.041 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:16.042 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:16.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2024-07-27T14:32:16.558 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:57 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:16.645 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:16.687 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:16.688 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:16.689 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:16.741 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-07-27T14:32:16.841 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:56 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:16.972 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 145 k 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:32:16.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:16.974 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 145 k 2024-07-27T14:32:16.974 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 435 k 2024-07-27T14:32:16.974 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:17.209 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-19.0.0-3573.g57fd71 620 kB/s | 145 kB 00:00 2024-07-27T14:32:17.209 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:17.209 INFO:teuthology.orchestra.run.smithi112.stdout:Total 615 kB/s | 145 kB 00:00 2024-07-27T14:32:17.210 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:17.217 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:17.217 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:17.275 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:17.275 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:17.284 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:58 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:17.411 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 3.6 M 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 32 k 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 16 k 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-27T14:32:17.413 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-07-27T14:32:17.414 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:17.415 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 8.7 M 2024-07-27T14:32:17.415 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 104 M 2024-07-27T14:32:17.415 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:17.420 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:17.476 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:17.501 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:17.501 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:17.501 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-27T14:32:17.501 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:18.234 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3573.g57 163 kB/s | 32 kB 00:00 2024-07-27T14:32:18.259 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3573.g57f 75 kB/s | 16 kB 00:00 2024-07-27T14:32:18.427 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3573.g57fd710 9.2 MB/s | 3.6 MB 00:00 2024-07-27T14:32:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103. 1/1 2024-07-27T14:32:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:18.450 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:18.451 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:18.451 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-07-27T14:32:18.527 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-07-27T14:32:18.552 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-07-27T14:32:18.636 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 24 MB/s | 2.0 MB 00:00 2024-07-27T14:32:18.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2024-07-27T14:32:18.669 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-07-27T14:32:18.720 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-07-27T14:32:18.753 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.9 MB/s | 188 kB 00:00 2024-07-27T14:32:18.778 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 282 kB/s | 31 kB 00:00 2024-07-27T14:32:18.804 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.2 MB/s | 863 kB 00:00 2024-07-27T14:32:18.807 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:18.807 INFO:teuthology.orchestra.run.smithi087.stdout:Total 6.2 MB/s | 8.7 MB 00:01 2024-07-27T14:32:18.855 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:18.872 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:18.872 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:19.050 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:19.050 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:19.222 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:19.310 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:19.354 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:19.355 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:32:19.355 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:19.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2024-07-27T14:32:19.577 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:19.646 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-07-27T14:32:19.690 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-07-27T14:32:19.736 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-07-27T14:32:19.767 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-07-27T14:32:19.877 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-07-27T14:32:19.958 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:20.006 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-07-27T14:32:20.086 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 3.6 M 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-27T14:32:20.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 32 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 16 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-07-27T14:32:20.089 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:20.090 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 8.7 M 2024-07-27T14:32:20.090 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 104 M 2024-07-27T14:32:20.090 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:20.092 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-07-27T14:32:20.136 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-07-27T14:32:20.163 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-07-27T14:32:20.195 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9 10/12 2024-07-27T14:32:20.788 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3573.g57f 87 kB/s | 16 kB 00:00 2024-07-27T14:32:20.805 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3573.g57 154 kB/s | 32 kB 00:00 2024-07-27T14:32:20.980 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3573.g57fd710 9.3 MB/s | 3.6 MB 00:00 2024-07-27T14:32:21.039 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-07-27T14:32:21.097 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-07-27T14:32:21.156 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-07-27T14:32:21.181 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-07-27T14:32:21.217 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.1 MB/s | 144 kB 00:00 2024-07-27T14:32:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el 11/12 2024-07-27T14:32:21.242 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-07-27T14:32:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 12/12 2024-07-27T14:32:21.284 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 16 MB/s | 2.0 MB 00:00 2024-07-27T14:32:21.318 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.5 MB/s | 863 kB 00:00 2024-07-27T14:32:21.343 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-07-27T14:32:21.347 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:21.347 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.9 MB/s | 8.7 MB 00:01 2024-07-27T14:32:21.394 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:21.411 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:21.411 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:21.599 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:21.600 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:22.119 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:22.209 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-07-27T14:32:22.245 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-07-27T14:32:22.291 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-07-27T14:32:22.322 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-07-27T14:32:22.344 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 12/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el 1/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 2/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9 3/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-07-27T14:32:22.345 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-07-27T14:32:22.437 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-07-27T14:32:22.565 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-27T14:32:22.656 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:22.657 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:22.658 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-07-27T14:32:22.702 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-07-27T14:32:22.730 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-07-27T14:32:22.778 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9 10/12 2024-07-27T14:32:22.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-07-27T14:32:23.443 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:23.570 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:23.571 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-07-27T14:32:23.571 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:23.571 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-07-27T14:32:23.571 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:23.571 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 7.4 M 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:23.572 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 38 M 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 209 M 2024-07-27T14:32:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:23.822 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el 11/12 2024-07-27T14:32:23.862 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 12/12 2024-07-27T14:32:24.367 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-35 15 MB/s | 7.4 MB 00:00 2024-07-27T14:32:24.450 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 394 kB/s | 33 kB 00:00 2024-07-27T14:32:24.509 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 312 kB/s | 195 kB 00:00 2024-07-27T14:32:24.584 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 244 kB/s | 18 kB 00:00 2024-07-27T14:32:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 604 kB/s | 35 kB 00:00 2024-07-27T14:32:24.693 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-07-27T14:32:24.718 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): libgfortran-11.4.1-3.el9.x86_64.rpm 962 kB/s | 803 kB 00:00 2024-07-27T14:32:24.743 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-07-27T14:32:24.785 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-07-27T14:32:24.877 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 20 MB/s | 4.7 MB 00:00 2024-07-27T14:32:25.119 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-07-27T14:32:25.123 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 12/12 2024-07-27T14:32:25.123 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el 1/12 2024-07-27T14:32:25.123 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noa 2/12 2024-07-27T14:32:25.123 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9 3/12 2024-07-27T14:32:25.123 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-07-27T14:32:25.124 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-07-27T14:32:25.186 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-07-27T14:32:25.188 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:25.188 INFO:teuthology.orchestra.run.smithi087.stdout:Total 24 MB/s | 38 MB 00:01 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-27T14:32:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:25.432 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:25.534 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:25.561 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:25.561 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:25.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2024-07-27T14:32:25.993 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:25.994 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:26.225 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:06 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:26.353 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:26.354 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-27T14:32:26.354 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 7.4 M 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-27T14:32:26.355 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-07-27T14:32:26.356 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:26.357 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 38 M 2024-07-27T14:32:26.357 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 209 M 2024-07-27T14:32:26.357 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:26.750 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 821 kB/s | 195 kB 00:00 2024-07-27T14:32:26.761 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:26.817 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 492 kB/s | 33 kB 00:00 2024-07-27T14:32:26.832 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-07-27T14:32:26.851 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-07-27T14:32:26.875 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-07-27T14:32:26.901 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-07-27T14:32:26.975 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-07-27T14:32:26.993 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-35 15 MB/s | 7.4 MB 00:00 2024-07-27T14:32:27.010 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-07-27T14:32:27.018 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 302 kB/s | 35 kB 00:00 2024-07-27T14:32:27.076 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-07-27T14:32:27.186 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-07-27T14:32:27.217 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-07-27T14:32:27.301 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-07-27T14:32:27.568 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-07-27T14:32:27.710 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-07-27T14:32:27.951 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-07-27T14:32:28.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-07-27T14:32:28.835 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.6 MB/s | 4.7 MB 00:01 2024-07-27T14:32:29.301 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-07-27T14:32:29.352 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.2 MB/s | 5.1 MB 00:02 2024-07-27T14:32:30.329 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-07-27T14:32:30.357 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 12/12 2024-07-27T14:32:32.394 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-07-27T14:32:32.395 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:32.395 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.3 MB/s | 38 MB 00:06 2024-07-27T14:32:32.749 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:32.776 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:32.776 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 12/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 1/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-07-27T14:32:32.868 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-07-27T14:32:32.869 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-07-27T14:32:33.218 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:33.218 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:33.266 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-07-27T14:32:33.266 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:33.266 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-27T14:32:33.267 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-27T14:32:33.268 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-27T14:32:33.269 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:33.269 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:33.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-07-27T14:32:33.977 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:34.063 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-07-27T14:32:34.108 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-07-27T14:32:34.211 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-07-27T14:32:34.243 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-07-27T14:32:34.255 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:15 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:34.383 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:34.384 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:34.384 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:34.384 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 49 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-27T14:32:34.385 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-07-27T14:32:34.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:34.387 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.2 M 2024-07-27T14:32:34.387 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-07-27T14:32:34.387 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:34.420 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-07-27T14:32:34.450 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-07-27T14:32:34.535 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-07-27T14:32:34.625 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 340 kB/s | 26 kB 00:00 2024-07-27T14:32:34.650 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 191 kB/s | 19 kB 00:00 2024-07-27T14:32:34.717 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.3 MB/s | 222 kB 00:00 2024-07-27T14:32:34.742 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-07-27T14:32:34.767 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): ceph-mgr-rook-19.0.0-3573.g57fd7103.el9 226 kB/s | 49 kB 00:00 2024-07-27T14:32:34.793 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-07-27T14:32:34.818 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-07-27T14:32:34.918 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-07-27T14:32:34.943 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 94 kB/s | 14 kB 00:00 2024-07-27T14:32:34.969 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-27T14:32:34.994 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-google-auth-2.32.0-1.el9.noarc 1.6 MB/s | 216 kB 00:00 2024-07-27T14:32:35.019 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-07-27T14:32:35.061 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-07-27T14:32:35.063 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:35.064 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-07-27T14:32:35.126 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:35.141 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:35.141 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:35.181 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-07-27T14:32:35.226 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:35.227 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-07-27T14:32:35.727 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-07-27T14:32:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-07-27T14:32:36.001 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-07-27T14:32:36.038 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-07-27T14:32:36.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-07-27T14:32:36.186 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-07-27T14:32:36.292 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-07-27T14:32:36.348 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-07-27T14:32:36.540 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-07-27T14:32:37.072 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-07-27T14:32:37.120 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-07-27T14:32:37.154 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-07-27T14:32:37.202 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-07-27T14:32:37.232 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 13/13 2024-07-27T14:32:37.568 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-07-27T14:32:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 12/12 2024-07-27T14:32:37.972 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 13/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 1/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-07-27T14:32:37.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-07-27T14:32:37.974 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-27T14:32:38.296 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:38.297 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:38.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-07-27T14:32:39.095 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:20 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:39.222 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:39.223 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:39.223 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:39.223 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 146 k 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-07-27T14:32:39.224 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:39.225 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.0 M 2024-07-27T14:32:39.225 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-07-27T14:32:39.225 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:39.596 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3573.g57fd7103.e 626 kB/s | 146 kB 00:00 2024-07-27T14:32:39.663 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-07-27T14:32:39.688 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-07-27T14:32:39.714 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-07-27T14:32:40.127 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 12/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7 1/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-07-27T14:32:40.128 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-07-27T14:32:40.330 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 257 kB/s | 248 kB 00:00 2024-07-27T14:32:40.559 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-07-27T14:32:40.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:40.559 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-27T14:32:40.560 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:40.561 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:41.032 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2024-07-27T14:32:41.589 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.7 MB/s | 6.0 MB 00:02 2024-07-27T14:32:41.591 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:41.592 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.0 MB/s | 7.0 MB 00:02 2024-07-27T14:32:41.592 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:21 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:41.664 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:41.676 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:41.677 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:41.722 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 49 k 2024-07-27T14:32:41.723 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-27T14:32:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:41.725 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.2 M 2024-07-27T14:32:41.726 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2024-07-27T14:32:41.726 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:41.778 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:41.779 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:42.088 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:42.149 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 26 kB 00:00 2024-07-27T14:32:42.175 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 19 kB 00:00 2024-07-27T14:32:42.199 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): ceph-mgr-rook-19.0.0-3573.g57fd7103.el9 255 kB/s | 49 kB 00:00 2024-07-27T14:32:42.245 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-07-27T14:32:42.292 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-07-27T14:32:42.298 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-07-27T14:32:42.317 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-07-27T14:32:42.367 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-07-27T14:32:42.392 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-07-27T14:32:42.443 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-07-27T14:32:42.501 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 107 kB/s | 14 kB 00:00 2024-07-27T14:32:42.535 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-07-27T14:32:42.560 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-google-auth-2.32.0-1.el9.noarc 1.5 MB/s | 216 kB 00:00 2024-07-27T14:32:42.585 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-07-27T14:32:42.619 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-07-27T14:32:42.621 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:42.621 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-07-27T14:32:42.683 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:42.698 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:42.698 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:42.785 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:42.786 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:42.815 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-07-27T14:32:42.909 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-07-27T14:32:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-07-27T14:32:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 6/6 2024-07-27T14:32:43.284 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:43.448 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-07-27T14:32:43.495 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-07-27T14:32:43.558 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-07-27T14:32:43.596 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-07-27T14:32:43.638 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-07-27T14:32:43.745 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-07-27T14:32:43.763 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 6/6 2024-07-27T14:32:43.763 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/6 2024-07-27T14:32:43.764 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-07-27T14:32:43.764 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-07-27T14:32:43.764 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-07-27T14:32:43.764 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-07-27T14:32:43.848 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-07-27T14:32:43.897 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:44.029 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:44.278 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-07-27T14:32:44.613 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-07-27T14:32:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-07-27T14:32:44.694 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-07-27T14:32:44.743 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-07-27T14:32:44.764 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 13/13 2024-07-27T14:32:44.829 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:25 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:44.957 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:44.957 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:44.957 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 897 k 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-07-27T14:32:44.958 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:44.959 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 977 k 2024-07-27T14:32:44.959 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.9 M 2024-07-27T14:32:44.959 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:45.197 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 643 kB/s | 80 kB 00:00 2024-07-27T14:32:45.381 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.0.0-3573.g57fd7103.el9.x86_ 2.8 MB/s | 897 kB 00:00 2024-07-27T14:32:45.381 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:45.381 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.3 MB/s | 977 kB 00:00 2024-07-27T14:32:45.393 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:45.414 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:45.414 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:45.493 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:45.493 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:45.530 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 13/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 1/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-07-27T14:32:45.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-07-27T14:32:45.683 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:45.816 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-07-27T14:32:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-27T14:32:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-27T14:32:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-27T14:32:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-27T14:32:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:45.866 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:45.891 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:32:46.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2024-07-27T14:32:46.533 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:32:46.533 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/2 2024-07-27T14:32:46.703 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:26 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:46.820 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-07-27T14:32:46.820 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:46.820 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:46.820 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-07-27T14:32:46.821 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:46.821 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:46.830 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:46.831 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:46.831 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 146 k 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:46.832 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:46.833 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-07-27T14:32:46.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:46.833 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 7.0 M 2024-07-27T14:32:46.833 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 33 M 2024-07-27T14:32:46.833 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:47.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-07-27T14:32:47.674 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:28 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:47.687 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3573.g57fd7103.e 660 kB/s | 146 kB 00:00 2024-07-27T14:32:47.807 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:47.807 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:47.807 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-07-27T14:32:47.807 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:47.807 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 274 k 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-07-27T14:32:47.808 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:47.809 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 351 k 2024-07-27T14:32:47.809 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.5 M 2024-07-27T14:32:47.809 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:47.814 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-07-27T14:32:47.846 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-07-27T14:32:47.871 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-07-27T14:32:48.049 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 442 kB/s | 77 kB 00:00 2024-07-27T14:32:48.105 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 388 kB/s | 248 kB 00:00 2024-07-27T14:32:48.124 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-volume-19.0.0-3573.g57fd7103.el9.no 1.1 MB/s | 274 kB 00:00 2024-07-27T14:32:48.125 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:48.125 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.1 MB/s | 351 kB 00:00 2024-07-27T14:32:48.136 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:48.146 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:48.146 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:48.178 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:48.178 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:48.326 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:48.405 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.4 MB/s | 6.0 MB 00:00 2024-07-27T14:32:48.406 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:48.407 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.4 MB/s | 7.0 MB 00:01 2024-07-27T14:32:48.475 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:48.486 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:48.487 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:48.512 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-07-27T14:32:48.551 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2/2 2024-07-27T14:32:48.567 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2/2 2024-07-27T14:32:48.567 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:48.567 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-27T14:32:48.567 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:48.585 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:48.585 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:48.879 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:49.062 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-07-27T14:32:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-07-27T14:32:49.244 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 1/2 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-27T14:32:49.522 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:49.523 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:49.635 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-07-27T14:32:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-07-27T14:32:49.789 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-07-27T14:32:49.820 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-07-27T14:32:49.826 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 6/6 2024-07-27T14:32:50.380 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:31 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:50.513 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 127 k 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:50.514 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:50.515 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:32:50.515 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:50.515 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 127 k 2024-07-27T14:32:50.515 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2024-07-27T14:32:50.515 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:50.628 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 6/6 2024-07-27T14:32:50.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 1/6 2024-07-27T14:32:50.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-07-27T14:32:50.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-07-27T14:32:50.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-07-27T14:32:50.629 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-07-27T14:32:50.724 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.0.0-3573.g57fd7103.el9.x86_64 606 kB/s | 127 kB 00:00 2024-07-27T14:32:50.724 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:50.725 INFO:teuthology.orchestra.run.smithi087.stdout:Total 602 kB/s | 127 kB 00:00 2024-07-27T14:32:50.725 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:50.730 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:50.730 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:50.793 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:50.793 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:50.888 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-07-27T14:32:50.888 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:50.889 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:50.889 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:50.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-27T14:32:50.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-27T14:32:50.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-27T14:32:50.890 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-27T14:32:50.890 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-27T14:32:50.890 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:50.890 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:50.940 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:51.039 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:51.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2024-07-27T14:32:51.501 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:51.690 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:31 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:51.743 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:51.817 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 897 k 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-27T14:32:51.818 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 977 k 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.9 M 2024-07-27T14:32:51.819 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:51.985 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 627 kB/s | 80 kB 00:00 2024-07-27T14:32:51.992 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-07-27T14:32:52.160 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-19.0.0-3573.g57fd7103.el9.x86_ 2.9 MB/s | 897 kB 00:00 2024-07-27T14:32:52.160 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:52.160 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.8 MB/s | 977 kB 00:00 2024-07-27T14:32:52.172 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:52.193 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:52.193 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:52.273 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:52.273 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:52.458 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:52.559 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:33 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:52.586 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-07-27T14:32:52.648 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:52.650 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:32:52.693 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:52.694 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:52.694 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:52.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-07-27T14:32:53.287 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:32:53.288 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/2 2024-07-27T14:32:53.300 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:34 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:53.428 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:53.429 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:53.429 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:32:53.429 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:53.429 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 32 k 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-07-27T14:32:53.430 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:53.559 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-07-27T14:32:53.560 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:53.560 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:53.560 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-07-27T14:32:53.560 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:53.560 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:53.623 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.0.0-3573.g57fd7103.el9.x86_6 165 kB/s | 32 kB 00:00 2024-07-27T14:32:53.624 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:53.624 INFO:teuthology.orchestra.run.smithi087.stdout:Total 164 kB/s | 32 kB 00:00 2024-07-27T14:32:53.624 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:32:53.627 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:32:53.627 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:32:53.648 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:32:53.648 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:32:53.728 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:32:53.845 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:53.870 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-volume 2024-07-27T14:32:54.229 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:54.425 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:34 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:54.450 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:54.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:54.451 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:32:54.451 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:54.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:54.451 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-27T14:32:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-3573.g57fd7103.el9 ceph-noarch 274 k 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-07-27T14:32:54.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:54.560 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 351 k 2024-07-27T14:32:54.560 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.5 M 2024-07-27T14:32:54.560 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:54.622 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-07-27T14:32:54.933 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): ceph-volume-19.0.0-3573.g57fd7103.el9.no 1.0 MB/s | 274 kB 00:00 2024-07-27T14:32:55.167 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 158 kB/s | 77 kB 00:00 2024-07-27T14:32:55.167 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:36 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:55.168 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:55.168 INFO:teuthology.orchestra.run.smithi112.stdout:Total 576 kB/s | 351 kB 00:00 2024-07-27T14:32:55.182 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:55.190 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:55.190 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:55.227 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:55.227 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:55.252 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:55.294 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:55.295 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:55.295 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:55.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-07-27T14:32:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:55.557 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-07-27T14:32:55.586 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2/2 2024-07-27T14:32:55.601 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2/2 2024-07-27T14:32:55.601 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:32:55.602 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-27T14:32:55.602 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:55.913 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:36 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:55.999 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:56.042 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:56.043 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:56.043 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:56.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-07-27T14:32:56.280 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 1/2 2024-07-27T14:32:56.603 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-07-27T14:32:56.603 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:56.603 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:56.604 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-3573.g57fd7103.el9.noarch 2024-07-27T14:32:56.604 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-27T14:32:56.604 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:56.604 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:56.656 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:56.741 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:56.784 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:56.785 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:56.785 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:56.846 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-07-27T14:32:56.898 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2024-07-27T14:32:57.394 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:57.461 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:57.480 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:57.522 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:57.523 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:57.523 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:57.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-07-27T14:32:57.592 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 127 k 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:57.593 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 127 k 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 456 k 2024-07-27T14:32:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:32:57.797 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-19.0.0-3573.g57fd7103.el9.x86_64 622 kB/s | 127 kB 00:00 2024-07-27T14:32:57.798 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:32:57.798 INFO:teuthology.orchestra.run.smithi112.stdout:Total 617 kB/s | 127 kB 00:00 2024-07-27T14:32:57.798 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:32:57.804 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:32:57.804 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:32:57.868 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:32:57.868 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:32:58.016 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:32:58.129 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:58.138 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:39 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:58.223 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:58.265 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:58.266 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:58.266 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:58.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-07-27T14:32:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:58.886 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:39 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:32:58.889 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:58.971 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:59.014 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:59.014 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:32:59.015 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:32:59.082 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-07-27T14:32:59.175 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2024-07-27T14:32:59.631 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:40 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:32:59.745 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:39 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:32:59.760 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:32:59.760 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:59.760 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 85 k 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2024-07-27T14:32:59.761 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:32:59.834 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:32:59.877 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:32:59.878 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:32:59.878 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:32:59.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2024-07-27T14:33:00.095 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.0.0-3573.g57fd7103.el9.x86_64.rpm 256 kB/s | 85 kB 00:00 2024-07-27T14:33:00.095 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:00.095 INFO:teuthology.orchestra.run.smithi087.stdout:Total 255 kB/s | 85 kB 00:00 2024-07-27T14:33:00.095 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:33:00.101 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:33:00.102 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:33:00.164 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:33:00.164 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:33:00.308 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:33:00.427 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:00.508 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:40 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:00.636 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 32 k 2024-07-27T14:33:00.637 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 32 k 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 142 k 2024-07-27T14:33:00.638 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:00.797 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-19.0.0-3573.g57fd7103.el9.x86_6 199 kB/s | 32 kB 00:00 2024-07-27T14:33:00.798 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:00.798 INFO:teuthology.orchestra.run.smithi112.stdout:Total 197 kB/s | 32 kB 00:00 2024-07-27T14:33:00.798 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:00.801 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:00.802 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:00.802 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:00.822 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:00.823 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:33:00.905 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:33:01.022 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:01.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:01.053 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:01.054 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:33:01.054 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:01.054 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:01.054 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:33:01.308 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-07-27T14:33:01.414 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:01.643 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:01.643 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:01.643 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:33:01.643 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:01.644 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:01.644 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:01.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2024-07-27T14:33:01.888 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:42 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:33:02.021 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.1 M 2024-07-27T14:33:02.022 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.1 M 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 13 M 2024-07-27T14:33:02.023 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:33:02.360 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:42 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:02.391 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.0.0-3573.g57fd7103.el9.x86_64.rpm 8.4 MB/s | 3.1 MB 00:00 2024-07-27T14:33:02.393 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:02.393 INFO:teuthology.orchestra.run.smithi087.stdout:Total 8.4 MB/s | 3.1 MB 00:00 2024-07-27T14:33:02.393 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:33:02.402 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:33:02.402 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:33:02.445 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:02.482 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:33:02.483 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:33:02.487 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:02.488 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:02.488 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:02.552 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2024-07-27T14:33:02.921 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:33:02.984 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-27T14:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T14:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T14:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:03.093 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:43 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:03.177 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:03.219 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:03.220 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:03.220 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:03.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2024-07-27T14:33:03.835 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:43 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:03.919 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:03.962 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:03.962 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:03.963 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:04.023 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2024-07-27T14:33:04.092 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:04.093 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:04.093 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:33:04.093 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:04.093 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:04.093 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:33:04.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-07-27T14:33:04.571 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:44 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:04.655 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:04.698 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:04.698 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:04.699 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:04.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2024-07-27T14:33:04.905 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:45 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:33:05.034 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:33:05.034 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 170 k 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 170 k 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 494 k 2024-07-27T14:33:05.035 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:33:05.315 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:45 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:05.400 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:05.444 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:05.445 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:05.445 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:05.469 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.0.0-3573.g57fd7103.el9.x86_64.rpm 393 kB/s | 170 kB 00:00 2024-07-27T14:33:05.469 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:05.470 INFO:teuthology.orchestra.run.smithi087.stdout:Total 392 kB/s | 170 kB 00:00 2024-07-27T14:33:05.470 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:33:05.478 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:33:05.478 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:33:05.516 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2024-07-27T14:33:05.540 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:33:05.541 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:33:05.691 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:33:05.784 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:06.065 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:46 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:06.151 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:33:06.195 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:06.196 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:33:06.196 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:06.210 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:06.257 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2024-07-27T14:33:06.457 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:06.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:06.458 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:33:06.458 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:06.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:06.458 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:33:06.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base-debuginfo 2024-07-27T14:33:06.805 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:46 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:06.934 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:06.935 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-07-27T14:33:06.935 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 85 k 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 85 k 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 231 k 2024-07-27T14:33:06.936 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:07.152 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-19.0.0-3573.g57fd7103.el9.x86_64.rpm 394 kB/s | 85 kB 00:00 2024-07-27T14:33:07.153 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:07.153 INFO:teuthology.orchestra.run.smithi112.stdout:Total 391 kB/s | 85 kB 00:00 2024-07-27T14:33:07.153 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:07.222 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:07.223 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:33:07.263 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:48 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:33:07.367 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:33:07.397 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:33:07.397 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:07.397 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:33:07.397 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 69 M 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 23 M 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-07-27T14:33:07.398 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:07.399 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 92 M 2024-07-27T14:33:07.399 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 504 M 2024-07-27T14:33:07.399 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:33:07.475 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:08.102 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:08.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2024-07-27T14:33:08.956 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:48 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:09.092 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 3.1 M 2024-07-27T14:33:09.093 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.1 M 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 13 M 2024-07-27T14:33:09.094 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:09.475 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-19.0.0-3573.g57fd7103.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-07-27T14:33:09.476 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:09.476 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-07-27T14:33:09.476 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:09.485 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:09.485 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:09.569 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:09.569 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:33:09.725 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): ceph-base-debuginfo-19.0.0-3573.g57fd710 30 MB/s | 69 MB 00:02 2024-07-27T14:33:09.942 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-debuginfo-19.0.0-3573.g57fd7103.el9 9.2 MB/s | 23 MB 00:02 2024-07-27T14:33:09.944 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:09.944 INFO:teuthology.orchestra.run.smithi087.stdout:Total 36 MB/s | 92 MB 00:02 2024-07-27T14:33:09.944 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:33:09.949 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:33:09.949 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:33:10.005 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:33:10.059 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T14:33:10.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:10.541 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:33:10.541 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:33:11.098 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:11.099 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:11.099 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:33:11.099 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:11.099 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:11.099 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:11.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2024-07-27T14:33:11.897 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:51 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:12.025 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 170 k 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:33:12.026 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:12.027 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 170 k 2024-07-27T14:33:12.027 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 494 k 2024-07-27T14:33:12.027 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:12.253 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-19.0.0-3573.g57fd7103.el9.x86_64.rpm 754 kB/s | 170 kB 00:00 2024-07-27T14:33:12.254 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:12.254 INFO:teuthology.orchestra.run.smithi112.stdout:Total 749 kB/s | 170 kB 00:00 2024-07-27T14:33:12.254 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:12.261 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:12.262 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:12.324 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:12.324 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:33:12.476 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:33:12.584 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:13.054 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:13.300 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:33:13.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:13.301 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:33:13.301 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:13.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:13.301 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:13.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base-debuginfo 2024-07-27T14:33:13.748 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:33:14.096 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:54 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:14.226 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 69 M 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 23 M 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:14.227 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 92 M 2024-07-27T14:33:14.228 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 504 M 2024-07-27T14:33:14.228 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:16.662 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): ceph-base-debuginfo-19.0.0-3573.g57fd710 28 MB/s | 69 MB 00:02 2024-07-27T14:33:17.270 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-debuginfo-19.0.0-3573.g57fd7103.el9 7.7 MB/s | 23 MB 00:03 2024-07-27T14:33:17.274 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:17.274 INFO:teuthology.orchestra.run.smithi112.stdout:Total 30 MB/s | 92 MB 00:03 2024-07-27T14:33:17.274 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:17.277 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:17.277 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:17.867 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:17.867 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:33:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/2 2024-07-27T14:33:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 2/2 2024-07-27T14:33:21.081 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:33:24.422 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 2/2 2024-07-27T14:33:24.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/2 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:24.655 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:33:24.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-common-debuginfo 2024-07-27T14:33:25.433 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:06 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:33:25.562 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 453 M 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 453 M 2024-07-27T14:33:25.563 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.8 G 2024-07-27T14:33:25.564 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:33:28.466 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/2 2024-07-27T14:33:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 2/2 2024-07-27T14:33:31.927 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 2/2 2024-07-27T14:33:31.928 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/2 2024-07-27T14:33:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2/2 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:32.156 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:33:32.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-common-debuginfo 2024-07-27T14:33:32.927 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:12 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:33:33.055 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:33:33.056 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:33.056 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:33:33.056 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:33.056 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:33:33.056 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 453 M 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 453 M 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.8 G 2024-07-27T14:33:33.057 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:33:37.491 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-common-debuginfo-19.0.0-3573.g57fd7103.el9 38 MB/s | 453 MB 00:11 2024-07-27T14:33:37.492 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:37.492 INFO:teuthology.orchestra.run.smithi087.stdout:Total 38 MB/s | 453 MB 00:11 2024-07-27T14:33:37.493 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:33:37.497 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:33:37.497 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:33:40.174 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:33:40.174 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:33:46.225 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-common-debuginfo-19.0.0-3573.g57fd7103.el9 34 MB/s | 453 MB 00:13 2024-07-27T14:33:46.226 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:33:46.227 INFO:teuthology.orchestra.run.smithi112.stdout:Total 34 MB/s | 453 MB 00:13 2024-07-27T14:33:46.227 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:33:46.231 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:33:46.231 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:33:48.909 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:33:48.909 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:34:26.073 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:34:26.169 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:33.408 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:33.629 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:34:33.869 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-07-27T14:34:34.426 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:15 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:34:34.478 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:34:34.566 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:34:34.567 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-07-27T14:34:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:34:34.567 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-07-27T14:34:34.567 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.5 M 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.5 M 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 6.6 M 2024-07-27T14:34:34.568 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:34:34.619 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:34.891 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-35 4.5 MB/s | 1.5 MB 00:00 2024-07-27T14:34:34.891 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:34:34.892 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.5 MB/s | 1.5 MB 00:00 2024-07-27T14:34:34.892 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:34:34.895 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:34:34.896 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:34:34.939 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:34:34.939 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:34:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:34:35.297 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:35.680 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:35.906 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:34:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:34:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:35.907 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:34:36.100 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw-debuginfo 2024-07-27T14:34:36.650 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:17 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:34:36.779 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:34:36.779 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 188 M 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 188 M 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 800 M 2024-07-27T14:34:36.780 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:34:42.552 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-radosgw-debuginfo-19.0.0-3573.g57fd7103.el 33 MB/s | 188 MB 00:05 2024-07-27T14:34:42.552 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:34:42.553 INFO:teuthology.orchestra.run.smithi087.stdout:Total 33 MB/s | 188 MB 00:05 2024-07-27T14:34:42.553 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:34:42.558 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:34:42.558 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:34:43.110 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:43.322 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:34:43.322 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:43.322 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:34:43.322 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:34:43.322 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:43.323 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:34:43.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-07-27T14:34:43.688 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:34:43.688 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:34:44.130 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:24 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:34:44.266 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:34:44.267 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-07-27T14:34:44.267 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:34:44.267 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.5 M 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.5 M 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 6.6 M 2024-07-27T14:34:44.268 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:34:44.550 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-35 5.2 MB/s | 1.5 MB 00:00 2024-07-27T14:34:44.550 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:34:44.551 INFO:teuthology.orchestra.run.smithi112.stdout:Total 5.1 MB/s | 1.5 MB 00:00 2024-07-27T14:34:44.551 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:34:44.554 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:34:44.554 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:34:44.599 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:34:44.599 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:34:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:34:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-3573. 1/1 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:45.587 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:34:45.771 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw-debuginfo 2024-07-27T14:34:46.314 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:26 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:34:46.441 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 188 M 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:34:46.442 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:34:46.443 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:34:46.443 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:34:46.443 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 188 M 2024-07-27T14:34:46.443 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 800 M 2024-07-27T14:34:46.443 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:34:52.177 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-radosgw-debuginfo-19.0.0-3573.g57fd7103.el 33 MB/s | 188 MB 00:05 2024-07-27T14:34:52.178 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:34:52.179 INFO:teuthology.orchestra.run.smithi112.stdout:Total 33 MB/s | 188 MB 00:05 2024-07-27T14:34:52.179 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:34:52.184 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:34:52.184 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:34:53.320 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:34:53.320 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:35:02.863 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:35:02.957 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:07.758 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:35:08.000 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:35:08.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test-debuginfo 2024-07-27T14:35:08.765 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:49 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:35:08.893 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:35:08.893 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 694 M 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 694 M 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 3.2 G 2024-07-27T14:35:08.894 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:35:12.695 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:35:12.789 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:17.749 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x 1/1 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:35:17.983 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:35:18.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test-debuginfo 2024-07-27T14:35:18.735 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:03:58 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:35:18.863 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 694 M 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:35:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:35:18.865 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 694 M 2024-07-27T14:35:18.865 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 3.2 G 2024-07-27T14:35:18.865 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:35:29.376 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-test-debuginfo-19.0.0-3573.g57fd7103.el9.x 34 MB/s | 694 MB 00:20 2024-07-27T14:35:29.377 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:35:29.377 INFO:teuthology.orchestra.run.smithi087.stdout:Total 34 MB/s | 694 MB 00:20 2024-07-27T14:35:29.378 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:35:29.384 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:35:29.384 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:35:33.484 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:35:33.485 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:35:39.527 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-test-debuginfo-19.0.0-3573.g57fd7103.el9.x 34 MB/s | 694 MB 00:20 2024-07-27T14:35:39.528 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:35:39.528 INFO:teuthology.orchestra.run.smithi112.stdout:Total 34 MB/s | 694 MB 00:20 2024-07-27T14:35:39.529 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:35:39.534 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:35:39.534 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:35:43.617 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:35:43.617 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:36:46.417 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:36:46.562 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:36:54.236 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:36:54.460 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:36:54.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:36:54.461 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:36:54.461 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:36:54.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:36:54.461 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:36:54.775 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base-debuginfo 2024-07-27T14:36:55.328 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:05:36 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:36:55.414 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:36:55.457 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:36:55.458 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-07-27T14:36:55.458 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:36:55.526 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-debuginfo 2024-07-27T14:36:56.078 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:05:37 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:36:56.206 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 20 M 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:36:56.207 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:36:56.208 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:36:56.208 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:36:56.208 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 20 M 2024-07-27T14:36:56.208 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 105 M 2024-07-27T14:36:56.208 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:36:56.479 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:36:56.983 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mgr-debuginfo-19.0.0-3573.g57fd7103.el9.x8 26 MB/s | 20 MB 00:00 2024-07-27T14:36:56.984 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:36:56.984 INFO:teuthology.orchestra.run.smithi087.stdout:Total 26 MB/s | 20 MB 00:00 2024-07-27T14:36:56.985 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:36:56.988 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:36:56.988 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:36:57.144 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:36:57.144 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:36:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:36:59.440 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:36:59.606 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:00.570 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:00.792 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:00.793 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:00.793 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:37:00.793 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:00.793 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:00.793 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:37:00.974 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mds-debuginfo 2024-07-27T14:37:01.523 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:05:42 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:37:01.653 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:37:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:01.654 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:37:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:37:01.654 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 53 M 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 53 M 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 195 M 2024-07-27T14:37:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:37:03.303 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mds-debuginfo-19.0.0-3573.g57fd7103.el9.x8 32 MB/s | 53 MB 00:01 2024-07-27T14:37:03.303 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:03.304 INFO:teuthology.orchestra.run.smithi087.stdout:Total 32 MB/s | 53 MB 00:01 2024-07-27T14:37:03.304 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:37:03.308 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:37:03.309 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:37:03.657 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:37:03.657 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:37:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:05.501 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:37:05.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base-debuginfo 2024-07-27T14:37:06.315 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:05:46 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:37:06.401 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 is already installed. 2024-07-27T14:37:06.444 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:37:06.444 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-07-27T14:37:06.444 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:37:06.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-debuginfo 2024-07-27T14:37:07.056 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:05:47 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:37:07.185 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 20 M 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:37:07.186 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:07.187 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 20 M 2024-07-27T14:37:07.187 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 105 M 2024-07-27T14:37:07.187 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:37:07.968 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-mgr-debuginfo-19.0.0-3573.g57fd7103.el9.x8 26 MB/s | 20 MB 00:00 2024-07-27T14:37:07.969 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:07.969 INFO:teuthology.orchestra.run.smithi112.stdout:Total 26 MB/s | 20 MB 00:00 2024-07-27T14:37:07.969 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:37:07.973 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:37:07.973 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:37:08.128 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:37:08.129 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:37:08.962 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:37:09.092 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:10.417 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:37:10.558 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:10.775 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:37:10.799 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:10.949 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mon-debuginfo 2024-07-27T14:37:11.499 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:05:52 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:37:11.628 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 99 M 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:11.629 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:37:11.630 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:11.630 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 99 M 2024-07-27T14:37:11.630 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 451 M 2024-07-27T14:37:11.630 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:37:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:12.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:12.028 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:12.028 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:37:12.028 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:12.028 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:12.028 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:37:12.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mds-debuginfo 2024-07-27T14:37:12.743 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:05:52 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:37:12.871 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:37:12.872 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:12.872 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:37:12.872 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 53 M 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 53 M 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 195 M 2024-07-27T14:37:12.873 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:37:14.413 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mon-debuginfo-19.0.0-3573.g57fd7103.el9.x8 36 MB/s | 99 MB 00:02 2024-07-27T14:37:14.414 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:14.415 INFO:teuthology.orchestra.run.smithi087.stdout:Total 36 MB/s | 99 MB 00:02 2024-07-27T14:37:14.415 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:37:14.419 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:37:14.419 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:37:15.030 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:37:15.031 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:37:15.487 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-mds-debuginfo-19.0.0-3573.g57fd7103.el9.x8 20 MB/s | 53 MB 00:02 2024-07-27T14:37:15.488 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:15.488 INFO:teuthology.orchestra.run.smithi112.stdout:Total 20 MB/s | 53 MB 00:02 2024-07-27T14:37:15.489 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:37:15.493 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:37:15.493 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:37:15.840 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:37:15.840 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:37:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:37:21.251 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:22.684 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:22.908 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:22.909 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:22.909 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:37:22.909 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:22.909 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:22.909 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:37:23.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mon-debuginfo 2024-07-27T14:37:23.632 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:06:03 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:37:23.759 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:37:23.760 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:37:23.760 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:23.760 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 99 M 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 99 M 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 451 M 2024-07-27T14:37:23.761 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:37:25.398 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:37:25.477 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:26.514 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-mon-debuginfo-19.0.0-3573.g57fd7103.el9.x8 36 MB/s | 99 MB 00:02 2024-07-27T14:37:26.515 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:26.516 INFO:teuthology.orchestra.run.smithi112.stdout:Total 36 MB/s | 99 MB 00:02 2024-07-27T14:37:26.516 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:37:26.520 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:37:26.520 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:37:27.135 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:37:27.135 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:37:28.368 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:28.618 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:37:28.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-osd-debuginfo 2024-07-27T14:37:29.346 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:06:10 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:37:29.474 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:37:29.475 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:29.475 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:37:29.475 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:29.475 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:37:29.475 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 289 M 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 289 M 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.2 G 2024-07-27T14:37:29.476 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:37:37.232 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-osd-debuginfo-19.0.0-3573.g57fd7103.el9.x8 37 MB/s | 289 MB 00:07 2024-07-27T14:37:37.232 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:37.233 INFO:teuthology.orchestra.run.smithi087.stdout:Total 37 MB/s | 289 MB 00:07 2024-07-27T14:37:37.233 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:37:37.237 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:37:37.237 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:37:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:37:37.554 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:38.956 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:37:38.956 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:37:40.471 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:40.708 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:37:40.709 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:40.709 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:37:40.709 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:37:40.709 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:40.709 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:37:40.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-osd-debuginfo 2024-07-27T14:37:41.452 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:06:21 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:37:41.580 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:37:41.580 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:41.580 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 289 M 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:37:41.581 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:37:41.582 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 289 M 2024-07-27T14:37:41.582 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.2 G 2024-07-27T14:37:41.582 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:37:49.141 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-osd-debuginfo-19.0.0-3573.g57fd7103.el9.x8 38 MB/s | 289 MB 00:07 2024-07-27T14:37:49.142 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:37:49.143 INFO:teuthology.orchestra.run.smithi112.stdout:Total 38 MB/s | 289 MB 00:07 2024-07-27T14:37:49.143 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:37:49.147 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:37:49.147 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:37:50.866 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:37:50.867 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:08.449 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:08.744 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:16.120 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:16.352 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:16.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse-debuginfo 2024-07-27T14:38:17.115 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:06:58 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:17.243 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 6.0 M 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:17.244 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:17.245 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:17.245 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.0 M 2024-07-27T14:38:17.245 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 29 M 2024-07-27T14:38:17.245 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:17.673 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-fuse-debuginfo-19.0.0-3573.g57fd7103.el9.x 14 MB/s | 6.0 MB 00:00 2024-07-27T14:38:17.674 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:17.674 INFO:teuthology.orchestra.run.smithi087.stdout:Total 14 MB/s | 6.0 MB 00:00 2024-07-27T14:38:17.675 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:17.678 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:17.678 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:17.748 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:17.748 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:18.493 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:18.588 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:19.167 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:19.417 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:19.418 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:19.418 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:19.418 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:19.418 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:19.418 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:19.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel-debuginfo 2024-07-27T14:38:20.140 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:01 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:20.266 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 109 k 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:20.267 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:20.268 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 109 k 2024-07-27T14:38:20.268 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 440 k 2024-07-27T14:38:20.268 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:20.453 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:20.484 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-debuginfo-19.0.0-3573.g57fd7103. 503 kB/s | 109 kB 00:00 2024-07-27T14:38:20.485 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:20.485 INFO:teuthology.orchestra.run.smithi087.stdout:Total 500 kB/s | 109 kB 00:00 2024-07-27T14:38:20.485 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:20.488 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:20.488 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:20.523 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:20.523 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:21.384 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:21.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2-debuginfo 2024-07-27T14:38:22.114 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:03 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:22.242 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:22.242 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:22.242 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:22.242 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:22.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:22.244 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.4 M 2024-07-27T14:38:22.244 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 25 M 2024-07-27T14:38:22.244 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:22.646 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs2-debuginfo-19.0.0-3573.g57fd7103.el9. 14 MB/s | 5.4 MB 00:00 2024-07-27T14:38:22.646 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:22.647 INFO:teuthology.orchestra.run.smithi087.stdout:Total 13 MB/s | 5.4 MB 00:00 2024-07-27T14:38:22.647 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:22.650 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:22.650 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:23.412 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:23.517 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:24.007 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:24.290 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:24.491 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2-debuginfo 2024-07-27T14:38:25.035 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:06 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:25.162 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 45 M 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:25.163 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 45 M 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 257 M 2024-07-27T14:38:25.164 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:26.590 INFO:teuthology.orchestra.run.smithi087.stdout:librados2-debuginfo-19.0.0-3573.g57fd7103.el9.x 32 MB/s | 45 MB 00:01 2024-07-27T14:38:26.591 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:26.592 INFO:teuthology.orchestra.run.smithi087.stdout:Total 32 MB/s | 45 MB 00:01 2024-07-27T14:38:26.592 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:26.596 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:26.596 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:26.902 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:26.902 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:27.928 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:28.162 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:28.162 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:28.163 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:28.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse-debuginfo 2024-07-27T14:38:28.924 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:08 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:29.053 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 6.0 M 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:29.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:29.055 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 6.0 M 2024-07-27T14:38:29.055 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 29 M 2024-07-27T14:38:29.055 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:29.474 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-fuse-debuginfo-19.0.0-3573.g57fd7103.el9.x 14 MB/s | 6.0 MB 00:00 2024-07-27T14:38:29.475 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:29.476 INFO:teuthology.orchestra.run.smithi112.stdout:Total 14 MB/s | 6.0 MB 00:00 2024-07-27T14:38:29.476 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:29.479 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:29.479 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:29.548 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:29.549 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:30.291 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:30.380 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:30.893 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:31.220 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:31.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel-debuginfo 2024-07-27T14:38:31.935 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:11 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:32.063 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 109 k 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 109 k 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 440 k 2024-07-27T14:38:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:32.080 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:32.163 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:32.294 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-debuginfo-19.0.0-3573.g57fd7103. 474 kB/s | 109 kB 00:00 2024-07-27T14:38:32.295 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:32.295 INFO:teuthology.orchestra.run.smithi112.stdout:Total 472 kB/s | 109 kB 00:00 2024-07-27T14:38:32.295 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:32.298 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:32.298 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:32.334 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:32.334 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:32.897 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:33.123 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:33.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2-debuginfo 2024-07-27T14:38:33.858 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:13 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:33.986 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:33.987 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:33.987 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:33.987 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 5.4 M 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 5.4 M 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 25 M 2024-07-27T14:38:33.988 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:34.014 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:34.240 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:34.397 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs2-debuginfo-19.0.0-3573.g57fd7103.el9. 13 MB/s | 5.4 MB 00:00 2024-07-27T14:38:34.398 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:34.398 INFO:teuthology.orchestra.run.smithi112.stdout:Total 13 MB/s | 5.4 MB 00:00 2024-07-27T14:38:34.399 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:34.402 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:34.402 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:34.469 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:34.469 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:34.582 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1-debuginfo 2024-07-27T14:38:35.128 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:16 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:35.162 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:35.254 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 52 M 2024-07-27T14:38:35.255 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 52 M 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 309 M 2024-07-27T14:38:35.256 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:35.740 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:38:35.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:35.973 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:35.974 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:35.974 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:36.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2-debuginfo 2024-07-27T14:38:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:16 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:36.835 INFO:teuthology.orchestra.run.smithi087.stdout:librbd1-debuginfo-19.0.0-3573.g57fd7103.el9.x86 33 MB/s | 52 MB 00:01 2024-07-27T14:38:36.835 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:36.836 INFO:teuthology.orchestra.run.smithi087.stdout:Total 33 MB/s | 52 MB 00:01 2024-07-27T14:38:36.836 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:36.840 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:36.840 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:36.862 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 45 M 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:36.863 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:36.864 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 45 M 2024-07-27T14:38:36.864 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 257 M 2024-07-27T14:38:36.864 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:37.184 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:37.185 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:38.267 INFO:teuthology.orchestra.run.smithi112.stdout:librados2-debuginfo-19.0.0-3573.g57fd7103.el9.x 32 MB/s | 45 MB 00:01 2024-07-27T14:38:38.268 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:38.268 INFO:teuthology.orchestra.run.smithi112.stdout:Total 32 MB/s | 45 MB 00:01 2024-07-27T14:38:38.268 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:38.272 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:38.273 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:38.577 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:38.577 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:43.189 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:43.273 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:43.725 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:43.815 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:45.364 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:45.614 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:45.614 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:45.615 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:45.615 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:45.615 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:45.615 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:45.656 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:45.822 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs-debuginfo 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_ 1/1 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:45.882 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:46.075 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1-debuginfo 2024-07-27T14:38:46.368 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:27 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:46.496 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:46.496 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:46.496 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 615 k 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 615 k 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.1 M 2024-07-27T14:38:46.497 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:46.624 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:26 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 52 M 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 52 M 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 309 M 2024-07-27T14:38:46.756 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:46.770 INFO:teuthology.orchestra.run.smithi087.stdout:python3-cephfs-debuginfo-19.0.0-3573.g57fd7103. 2.2 MB/s | 615 kB 00:00 2024-07-27T14:38:46.770 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:46.771 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.2 MB/s | 615 kB 00:00 2024-07-27T14:38:46.771 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:46.774 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:46.774 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:46.813 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:46.813 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:46.983 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:47.092 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:47.475 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:47.704 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:47.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados-debuginfo 2024-07-27T14:38:48.337 INFO:teuthology.orchestra.run.smithi112.stdout:librbd1-debuginfo-19.0.0-3573.g57fd7103.el9.x86 33 MB/s | 52 MB 00:01 2024-07-27T14:38:48.337 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:48.338 INFO:teuthology.orchestra.run.smithi112.stdout:Total 33 MB/s | 52 MB 00:01 2024-07-27T14:38:48.338 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:48.342 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:48.342 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:48.440 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:29 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:48.569 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 769 k 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:48.570 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:48.571 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:48.571 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:48.571 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 769 k 2024-07-27T14:38:48.571 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.7 M 2024-07-27T14:38:48.571 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:48.688 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:48.688 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:48.865 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rados-debuginfo-19.0.0-3573.g57fd7103.e 2.6 MB/s | 769 kB 00:00 2024-07-27T14:38:48.866 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:48.866 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.5 MB/s | 769 kB 00:00 2024-07-27T14:38:48.866 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:48.870 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:48.870 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:48.910 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:48.910 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:49.090 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:49.189 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:38:49.537 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:38:49.771 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:38:49.771 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:49.772 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:49.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:49.772 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:49.772 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:49.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd-debuginfo 2024-07-27T14:38:50.510 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:31 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:50.640 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.1 M 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:50.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.1 M 2024-07-27T14:38:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 4.3 M 2024-07-27T14:38:50.642 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:50.926 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rbd-debuginfo-19.0.0-3573.g57fd7103.el9 4.0 MB/s | 1.1 MB 00:00 2024-07-27T14:38:50.927 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:50.927 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.0 MB/s | 1.1 MB 00:00 2024-07-27T14:38:50.927 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:50.930 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:50.931 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:50.973 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:50.973 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:51.194 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:51.299 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:51.684 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:51.920 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:51.921 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:51.921 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:51.921 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:51.921 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:51.921 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:52.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw-debuginfo 2024-07-27T14:38:52.664 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:33 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:52.791 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:52.791 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:52.791 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 337 k 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 337 k 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.0 M 2024-07-27T14:38:52.792 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:53.046 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rgw-debuginfo-19.0.0-3573.g57fd7103.el9 1.3 MB/s | 337 kB 00:00 2024-07-27T14:38:53.047 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:53.047 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-07-27T14:38:53.047 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:53.050 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:53.050 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:53.086 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:53.087 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:53.220 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:53.326 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:53.651 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:53.878 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:38:53.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:53.878 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:53.878 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:53.879 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:53.879 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:54.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse-debuginfo 2024-07-27T14:38:54.621 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:35 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:54.692 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:54.749 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:54.750 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:54.750 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 516 k 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 516 k 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.6 M 2024-07-27T14:38:54.751 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:54.783 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:55.008 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-debuginfo-19.0.0-3573.g57fd7103.el9.x8 2.0 MB/s | 516 kB 00:00 2024-07-27T14:38:55.008 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:55.008 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.0 MB/s | 516 kB 00:00 2024-07-27T14:38:55.009 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:55.012 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:55.012 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:55.049 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:55.050 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:55.210 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:38:55.314 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:55.661 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:55.886 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:38:56.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror-debuginfo 2024-07-27T14:38:56.622 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:37 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:38:56.750 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:38:56.751 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:56.751 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:38:56.751 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 52 M 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 52 M 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 299 M 2024-07-27T14:38:56.752 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:38:56.957 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:38:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:57.179 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:57.180 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:57.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs-debuginfo 2024-07-27T14:38:57.917 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:37 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:38:58.045 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:38:58.046 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:58.046 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:38:58.046 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 615 k 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 615 k 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.1 M 2024-07-27T14:38:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:38:58.339 INFO:teuthology.orchestra.run.smithi112.stdout:python3-cephfs-debuginfo-19.0.0-3573.g57fd7103. 2.1 MB/s | 615 kB 00:00 2024-07-27T14:38:58.340 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:58.340 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.0 MB/s | 615 kB 00:00 2024-07-27T14:38:58.340 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:38:58.343 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:38:58.343 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:38:58.344 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-debuginfo-19.0.0-3573.g57fd7103.el9. 32 MB/s | 52 MB 00:01 2024-07-27T14:38:58.345 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:38:58.345 INFO:teuthology.orchestra.run.smithi087.stdout:Total 32 MB/s | 52 MB 00:01 2024-07-27T14:38:58.346 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:38:58.349 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:38:58.350 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:38:58.382 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:38:58.382 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:38:58.548 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:38:58.637 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:58.688 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:38:58.688 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:38:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:59.204 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9 1/1 2024-07-27T14:38:59.205 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:59.205 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:38:59.205 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:38:59.205 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:38:59.205 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:38:59.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados-debuginfo 2024-07-27T14:38:59.979 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:39 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:00.106 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 769 k 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:00.107 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:00.108 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:00.108 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 769 k 2024-07-27T14:39:00.108 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.7 M 2024-07-27T14:39:00.108 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:00.401 INFO:teuthology.orchestra.run.smithi112.stdout:python3-rados-debuginfo-19.0.0-3573.g57fd7103.e 2.6 MB/s | 769 kB 00:00 2024-07-27T14:39:00.402 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:00.402 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.6 MB/s | 769 kB 00:00 2024-07-27T14:39:00.402 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:00.405 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:00.405 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:00.444 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:00.444 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:00.621 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:00.724 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:39:01.090 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:39:01.320 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9. 1/1 2024-07-27T14:39:01.321 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:01.321 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:01.321 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:01.321 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:01.321 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:01.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd-debuginfo 2024-07-27T14:39:02.066 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:42 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:02.194 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.1 M 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.1 M 2024-07-27T14:39:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 4.3 M 2024-07-27T14:39:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:02.495 INFO:teuthology.orchestra.run.smithi112.stdout:python3-rbd-debuginfo-19.0.0-3573.g57fd7103.el9 3.8 MB/s | 1.1 MB 00:00 2024-07-27T14:39:02.495 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:02.496 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2024-07-27T14:39:02.496 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:02.499 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:02.500 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:02.540 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:02.541 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:02.759 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:02.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:03.257 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:03.479 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:03.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw-debuginfo 2024-07-27T14:39:04.237 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:44 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:04.367 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 337 k 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:04.369 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:04.369 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:04.369 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 337 k 2024-07-27T14:39:04.369 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.0 M 2024-07-27T14:39:04.369 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:04.603 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:39:04.640 INFO:teuthology.orchestra.run.smithi112.stdout:python3-rgw-debuginfo-19.0.0-3573.g57fd7103.el9 1.2 MB/s | 337 kB 00:00 2024-07-27T14:39:04.640 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:04.641 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.2 MB/s | 337 kB 00:00 2024-07-27T14:39:04.641 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:04.644 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:04.644 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:04.681 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:04.681 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:04.708 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:04.819 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:04.921 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:05.282 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x8 1/1 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:05.523 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:05.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse-debuginfo 2024-07-27T14:39:06.257 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:46 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:06.385 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 516 k 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:06.386 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 516 k 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.6 M 2024-07-27T14:39:06.387 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:06.643 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-debuginfo-19.0.0-3573.g57fd7103.el9.x8 2.0 MB/s | 516 kB 00:00 2024-07-27T14:39:06.643 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:06.644 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.0 MB/s | 516 kB 00:00 2024-07-27T14:39:06.644 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:06.647 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:06.647 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:06.687 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:06.687 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:06.752 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:39:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:06.978 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:39:06.978 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:06.978 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:06.978 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:39:07.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd-debuginfo 2024-07-27T14:39:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:39:07.538 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_6 1/1 2024-07-27T14:39:07.539 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:07.539 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:07.539 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:07.539 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:07.539 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:07.715 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:07:48 ago on Sat 27 Jul 2024 02:31:19 PM UTC. 2024-07-27T14:39:07.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror-debuginfo 2024-07-27T14:39:07.845 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-07-27T14:39:07.845 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:39:07.845 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.3 M 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.3 M 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 5.7 M 2024-07-27T14:39:07.846 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-07-27T14:39:08.147 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-debuginfo-19.0.0-3573.g57fd7103.el9.x86 4.3 MB/s | 1.3 MB 00:00 2024-07-27T14:39:08.147 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:08.147 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.3 MB/s | 1.3 MB 00:00 2024-07-27T14:39:08.147 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-07-27T14:39:08.150 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-07-27T14:39:08.150 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-07-27T14:39:08.192 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-07-27T14:39:08.193 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-07-27T14:39:08.262 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:48 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:08.390 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 52 M 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:08.391 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:08.392 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:08.392 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 52 M 2024-07-27T14:39:08.392 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 299 M 2024-07-27T14:39:08.392 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:08.430 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-07-27T14:39:08.535 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:08.910 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:09.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:09.143 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:09.143 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-07-27T14:39:09.144 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:09.144 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:09.144 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-07-27T14:39:09.332 DEBUG:teuthology.parallel:result is None 2024-07-27T14:39:09.963 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-debuginfo-19.0.0-3573.g57fd7103.el9. 33 MB/s | 52 MB 00:01 2024-07-27T14:39:09.964 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:09.964 INFO:teuthology.orchestra.run.smithi112.stdout:Total 33 MB/s | 52 MB 00:01 2024-07-27T14:39:09.964 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:09.968 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:09.969 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:10.307 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:10.307 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:16.212 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:16.317 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86 1/1 2024-07-27T14:39:18.720 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:18.720 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:18.721 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:18.721 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:18.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd-debuginfo 2024-07-27T14:39:19.465 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:07:59 ago on Sat 27 Jul 2024 02:31:20 PM UTC. 2024-07-27T14:39:19.592 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-27T14:39:19.593 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:19.593 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-07-27T14:39:19.593 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:19.593 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-27T14:39:19.593 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-3573.g57fd7103.el9 ceph 1.3 M 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.3 M 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 5.7 M 2024-07-27T14:39:19.594 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-27T14:39:19.883 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-debuginfo-19.0.0-3573.g57fd7103.el9.x86 4.5 MB/s | 1.3 MB 00:00 2024-07-27T14:39:19.883 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-27T14:39:19.883 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.4 MB/s | 1.3 MB 00:00 2024-07-27T14:39:19.883 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-27T14:39:19.886 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-27T14:39:19.886 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-27T14:39:19.929 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-27T14:39:19.929 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-27T14:39:20.170 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-27T14:39:20.255 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:20.625 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:20.859 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 1/1 2024-07-27T14:39:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:20.860 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-27T14:39:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-debuginfo-2:19.0.0-3573.g57fd7103.el9.x86_64 2024-07-27T14:39:20.860 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:20.860 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-27T14:39:21.049 DEBUG:teuthology.parallel:result is None 2024-07-27T14:39:21.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:39:21.195 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-27T14:39:21.222 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-3573.g57fd7103.el9 2024-07-27T14:39:21.223 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3573.g57fd7103.el9 2024-07-27T14:39:21.223 INFO:teuthology.task.install:The correct ceph version 19.0.0-3573.g57fd7103 is installed. 2024-07-27T14:39:21.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:39:21.366 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-27T14:39:21.387 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-3573.g57fd7103.el9 2024-07-27T14:39:21.387 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3573.g57fd7103.el9 2024-07-27T14:39:21.388 INFO:teuthology.task.install:The correct ceph version 19.0.0-3573.g57fd7103 is installed. 2024-07-27T14:39:21.389 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-27T14:39:21.390 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:21.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-27T14:39:21.420 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:21.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-27T14:39:21.453 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-27T14:39:21.454 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:21.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-07-27T14:39:21.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-27T14:39:21.555 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:21.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-07-27T14:39:21.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-27T14:39:21.646 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-27T14:39:21.646 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:21.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-27T14:39:21.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-27T14:39:21.744 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:21.744 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-27T14:39:21.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-27T14:39:21.832 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-27T14:39:21.832 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:21.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-07-27T14:39:21.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-27T14:39:21.926 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:21.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-07-27T14:39:21.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-27T14:39:22.015 INFO:teuthology.run_tasks:Running task ceph... 2024-07-27T14:39:22.092 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-07-27T14:39:22.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-07-27T14:39:22.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-07-27T14:39:22.119 INFO:tasks.ceph:Disabling ceph logrotate... 2024-07-27T14:39:22.120 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-27T14:39:22.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-27T14:39:22.187 INFO:tasks.ceph:Creating extra log directories... 2024-07-27T14:39:22.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-27T14:39:22.228 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-27T14:39:22.264 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-07-27T14:39:22.264 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-07-27T14:39:22.264 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677', 'branch': 'squid', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7811677', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 2398, 'sha1': '57fd7103c78964675e1941b32d58041fb7c15af7', 'sleep_before_teardown': 0, 'subset': '24756/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '57fd7103c78964675e1941b32d58041fb7c15af7', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMN4uEnQYoPpP0A8dkQqVddzKoizfQbMSRV+vQm76Ua3NO0mr2a2BRH7I39sCOwy7G1pIkGDNDIKxvUFybPV0QE=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG/IfKsdn396O83xVDh4jpveUIu3IKKnftDh7EuLiTTfvVOFur/rptbxN+zbFKJ6gi5tCAuUDFv5zuBicf1eFk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e2a3aea75254d002e088a3ceee2a4fe2e292e3e', 'timestamp': '2024-07-21_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525'} 2024-07-27T14:39:22.265 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-27T14:39:22.295 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-27T14:39:22.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-07-27T14:39:22.351 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-07-27T14:39:22.378 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:22.378 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-07-27T14:39:22.429 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-27T14:39:22.430 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-07-27 14:33:03.259920455 +0000 2024-07-27T14:39:22.485 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-07-27 14:30:18.167586295 +0000 2024-07-27T14:39:22.486 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-07-27 14:30:18.167586295 +0000 2024-07-27T14:39:22.486 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-07-27 14:30:18.167586295 +0000 2024-07-27T14:39:22.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-27T14:39:22.548 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-07-27T14:39:22.548 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-07-27T14:39:22.548 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000235376 s, 2.2 MB/s 2024-07-27T14:39:22.549 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-27T14:39:22.605 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-07-27T14:39:22.660 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-07-27 14:33:03.260920433 +0000 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-07-27 14:30:18.539577432 +0000 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-07-27 14:30:18.539577432 +0000 2024-07-27T14:39:22.661 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-07-27 14:30:18.539577432 +0000 2024-07-27T14:39:22.662 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-27T14:39:22.724 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-07-27T14:39:22.724 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-07-27T14:39:22.724 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000186173 s, 2.8 MB/s 2024-07-27T14:39:22.725 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-27T14:39:22.781 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-07-27T14:39:22.836 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-27T14:39:22.836 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:22.836 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-07-27 14:33:03.260920433 +0000 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-07-27 14:30:18.875569427 +0000 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-07-27 14:30:18.875569427 +0000 2024-07-27T14:39:22.837 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-07-27 14:30:18.875569427 +0000 2024-07-27T14:39:22.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-27T14:39:22.900 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-07-27T14:39:22.900 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-07-27T14:39:22.900 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000184042 s, 2.8 MB/s 2024-07-27T14:39:22.901 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-27T14:39:22.957 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:23.012 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-07-27 14:33:03.260920433 +0000 2024-07-27T14:39:23.013 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-07-27 14:30:19.222561160 +0000 2024-07-27T14:39:23.013 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-07-27 14:30:19.222561160 +0000 2024-07-27T14:39:23.013 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-07-27 14:30:19.222561160 +0000 2024-07-27T14:39:23.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-27T14:39:23.074 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-07-27T14:39:23.075 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-07-27T14:39:23.075 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000209203 s, 2.4 MB/s 2024-07-27T14:39:23.076 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-27T14:39:23.132 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:23.132 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:23.132 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-07-27T14:39:23.149 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-27T14:39:23.149 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-27 14:33:10.333016843 +0000 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-27 14:30:15.505248720 +0000 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-27 14:30:15.505248720 +0000 2024-07-27T14:39:23.206 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-27 14:30:15.505248720 +0000 2024-07-27T14:39:23.206 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-27T14:39:23.271 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-27T14:39:23.271 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-27T14:39:23.271 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000193891 s, 2.6 MB/s 2024-07-27T14:39:23.272 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-27T14:39:23.329 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-07-27T14:39:23.385 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-27T14:39:23.385 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-27 14:33:10.334016816 +0000 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-27 14:30:15.842239834 +0000 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-27 14:30:15.842239834 +0000 2024-07-27T14:39:23.386 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-27 14:30:15.842239834 +0000 2024-07-27T14:39:23.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-27T14:39:23.450 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-27T14:39:23.450 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-27T14:39:23.450 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000202164 s, 2.5 MB/s 2024-07-27T14:39:23.452 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-27T14:39:23.508 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-27 14:33:10.334016816 +0000 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-27 14:30:16.168231237 +0000 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-27 14:30:16.168231237 +0000 2024-07-27T14:39:23.568 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-27 14:30:16.168231237 +0000 2024-07-27T14:39:23.569 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-27T14:39:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-27T14:39:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-27T14:39:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000195337 s, 2.6 MB/s 2024-07-27T14:39:23.632 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-27T14:39:23.689 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-07-27T14:39:23.744 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-27 14:33:10.334016816 +0000 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-27 14:30:16.514222112 +0000 2024-07-27T14:39:23.745 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-27 14:30:16.514222112 +0000 2024-07-27T14:39:23.746 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-27 14:30:16.514222112 +0000 2024-07-27T14:39:23.746 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-27T14:39:23.811 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-27T14:39:23.811 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-27T14:39:23.811 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000320922 s, 1.6 MB/s 2024-07-27T14:39:23.812 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-27T14:39:23.867 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:23.867 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-27T14:39:23.867 INFO:tasks.ceph:Generating config... 2024-07-27T14:39:23.868 INFO:tasks.ceph:[client] debug monc = 20 2024-07-27T14:39:23.869 INFO:tasks.ceph:[client] debug ms = 1 2024-07-27T14:39:23.869 INFO:tasks.ceph:[client] debug objecter = 20 2024-07-27T14:39:23.869 INFO:tasks.ceph:[client] debug rados = 20 2024-07-27T14:39:23.869 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-07-27T14:39:23.869 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-07-27T14:39:23.869 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-07-27T14:39:23.869 INFO:tasks.ceph:[global] ms type = async 2024-07-27T14:39:23.869 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-07-27T14:39:23.869 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-07-27T14:39:23.869 INFO:tasks.ceph:[mgr] debug ms = 1 2024-07-27T14:39:23.869 INFO:tasks.ceph:[mon] debug mon = 20 2024-07-27T14:39:23.869 INFO:tasks.ceph:[mon] debug ms = 1 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] debug paxos = 20 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-07-27T14:39:23.870 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-07-27T14:39:23.870 INFO:tasks.ceph:[osd] bdev async discard = True 2024-07-27T14:39:23.870 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-07-27T14:39:23.870 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-07-27T14:39:23.870 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug monc = 20 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug ms = 1 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug osd = 20 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] osd class default list = * 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] osd class load list = * 2024-07-27T14:39:23.871 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-07-27T14:39:23.872 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-07-27T14:39:23.872 INFO:tasks.ceph:Setting up mon.a... 2024-07-27T14:39:23.873 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-07-27T14:39:23.928 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2024-07-27T14:39:23.931 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-07-27T14:39:23.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-27T14:39:24.043 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.87'), ('mon.c', '[v2:172.21.15.87:3301,v1:172.21.15.87:6790]'), ('mon.b', '172.21.15.112')] 2024-07-27T14:39:24.043 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87,[v2:172.21.15.87:3301,v1:172.21.15.87:6790],172.21.15.112', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-07-27T14:39:24.044 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:24.044 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-07-27T14:39:24.098 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --addv c '[v2:172.21.15.87:3301,v1:172.21.15.87:6790]' --add b 172.21.15.112 --print /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 17 (quincy) 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:39:24.184 INFO:teuthology.orchestra.run.smithi087.stdout:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:39:24.185 INFO:teuthology.orchestra.run.smithi087.stdout:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:39:24.185 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-07-27T14:39:24.186 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-07-27T14:39:24.201 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a5c605d8-a191-47d5-9a8c-d25753c56cf8... 2024-07-27T14:39:24.201 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-27T14:39:24.244 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-07-27T14:39:24.284 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-07-27T14:39:24.285 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-07-27T14:39:24.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = "172.21.15.87,[v2:172.21.15.87:3301,v1:172.21.15.87:6790],172.21.15.112" 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: mon election default strategy = 1 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: ms type = async 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: osd heartbeat grace = 80 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:39:24.287 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-07-27T14:39:24.288 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.289 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = bitmap 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore zero block detection = True 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-07-27T14:39:24.290 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd class default list = * 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd class load list = * 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd client watch timeout = 120 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify cached snaps = True 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify missing on start = True 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd fast shutdown = False 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock override recovery settings = True 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue = debug_random 2024-07-27T14:39:24.291 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue cut off = debug_random 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-07-27T14:39:24.292 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.293 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd crush smoke test = False 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-07-27T14:39:24.294 INFO:teuthology.orchestra.run.smithi087.stdout: mon scrub interval = 300 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on pool no app = False 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: debug objecter = 20 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout: debug rados = 20 2024-07-27T14:39:24.295 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-07-27T14:39:24.296 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.c] 2024-07-27T14:39:24.296 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.b] 2024-07-27T14:39:24.296 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-07-27T14:39:24.296 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-07-27T14:39:24.296 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-07-27T14:39:24.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-07-27T14:39:24.298 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "172.21.15.87,[v2:172.21.15.87:3301,v1:172.21.15.87:6790],172.21.15.112" 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-07-27T14:39:24.299 INFO:teuthology.orchestra.run.smithi112.stdout: mon election default strategy = 1 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 5000 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: ms type = async 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd heartbeat grace = 80 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.300 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.301 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bdev async discard = True 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bdev enable discard = True 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore allocator = bitmap 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore zero block detection = True 2024-07-27T14:39:24.302 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 1 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 1 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 10 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd class default list = * 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd class load list = * 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd client watch timeout = 120 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify cached snaps = True 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify missing on start = True 2024-07-27T14:39:24.303 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd fast shutdown = False 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock override recovery settings = True 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue = debug_random 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue cut off = debug_random 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: osd shutdown pgref assert = True 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-07-27T14:39:24.304 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-07-27T14:39:24.305 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon min osdmap epochs = 50 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd crush smoke test = False 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune interval = 2 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune min = 15 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune txsize = 2 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub interval = 300 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: paxos service trim min = 10 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-07-27T14:39:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout: debug objecter = 20 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout: debug rados = 20 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2024-07-27T14:39:24.308 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2024-07-27T14:39:24.309 INFO:tasks.ceph:Creating admin key on mon.a... 2024-07-27T14:39:24.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-07-27T14:39:24.403 INFO:tasks.ceph:Copying monmap to all nodes... 2024-07-27T14:39:24.403 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:24.403 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-07-27T14:39:24.419 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:24.419 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-07-27T14:39:24.474 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:39:24.474 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:24.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-27T14:39:24.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-27T14:39:24.549 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:24.549 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:24.605 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:39:24.605 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:24.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-27T14:39:24.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-27T14:39:24.640 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:24.641 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:24.696 INFO:tasks.ceph:Setting up mon nodes... 2024-07-27T14:39:24.696 INFO:tasks.ceph:Setting up mgr nodes... 2024-07-27T14:39:24.697 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-07-27T14:39:24.747 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-07-27T14:39:24.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-07-27T14:39:24.823 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-07-27T14:39:24.826 INFO:tasks.ceph:Setting up mds nodes... 2024-07-27T14:39:24.826 INFO:tasks.ceph_client:Setting up client nodes... 2024-07-27T14:39:24.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-27T14:39:24.867 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-07-27T14:39:24.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-07-27T14:39:24.921 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-07-27T14:39:24.935 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-07-27T14:39:24.936 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-27T14:39:24.936 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-07-27T14:39:24.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:24.967 INFO:tasks.ceph:role: osd.0 2024-07-27T14:39:24.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:39:24.967 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-27T14:39:25.039 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:25.039 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:25.039 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:25.040 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:25.564 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-07-27T14:39:25.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-07-27T14:39:25.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-07-27T14:39:25.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-07-27T14:39:25.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-07-27T14:39:25.801 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:25.801 INFO:tasks.ceph:role: osd.1 2024-07-27T14:39:25.801 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:39:25.801 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-27T14:39:25.873 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:25.874 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:26.326 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-07-27T14:39:26.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-07-27T14:39:26.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-07-27T14:39:26.413 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-07-27T14:39:26.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-07-27T14:39:26.507 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:26.507 INFO:tasks.ceph:role: osd.2 2024-07-27T14:39:26.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:39:26.508 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:26.579 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:26.580 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:26.580 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:26.580 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:26.580 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:27.081 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-07-27T14:39:27.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-07-27T14:39:27.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-07-27T14:39:27.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-07-27T14:39:27.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-07-27T14:39:27.319 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:27.319 INFO:tasks.ceph:role: osd.3 2024-07-27T14:39:27.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T14:39:27.320 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-27T14:39:27.392 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:27.392 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:27.392 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:27.392 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:27.392 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:27.393 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:27.393 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:27.393 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:27.393 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:27.393 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:27.925 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-07-27T14:39:27.927 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-07-27T14:39:27.927 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-07-27T14:39:28.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-07-27T14:39:28.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:28.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.213+0000 7f554d744640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-07-27T14:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.214+0000 7f554d744640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-07-27T14:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.214+0000 7f554d744640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-27T14:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.214+0000 7f554d744640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-27T14:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.214+0000 7f554d744640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-27T14:39:28.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:28.215+0000 7f554d744640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-07-27T14:39:31.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-07-27T14:39:31.088 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:31.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.176+0000 7f6189814640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-07-27T14:39:31.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.177+0000 7f6189814640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-07-27T14:39:31.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.177+0000 7f6189814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-27T14:39:31.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.177+0000 7f6189814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-27T14:39:31.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.177+0000 7f6189814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-27T14:39:31.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:31.177+0000 7f6189814640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-07-27T14:39:34.029 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-07-27T14:39:34.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.148+0000 7f6f21b99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-07-27T14:39:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.148+0000 7f6f21b99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-07-27T14:39:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.148+0000 7f6f21b99640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-27T14:39:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.148+0000 7f6f21b99640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-27T14:39:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.148+0000 7f6f21b99640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-27T14:39:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:34.149+0000 7f6f21b99640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-07-27T14:39:36.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-07-27T14:39:37.027 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:37.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.117+0000 7f88a220d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-07-27T14:39:37.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.117+0000 7f88a220d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-07-27T14:39:37.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.117+0000 7f88a220d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-27T14:39:37.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.117+0000 7f88a220d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-27T14:39:37.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.117+0000 7f88a220d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-27T14:39:37.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:37.118+0000 7f88a220d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-07-27T14:39:39.961 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-07-27T14:39:39.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-07-27T14:39:40.018 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:40.018 INFO:tasks.ceph:role: osd.4 2024-07-27T14:39:40.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:39:40.019 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-27T14:39:40.093 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:40.093 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:40.094 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:40.598 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-07-27T14:39:40.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-07-27T14:39:40.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-07-27T14:39:40.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-07-27T14:39:40.782 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-07-27T14:39:40.851 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:40.851 INFO:tasks.ceph:role: osd.5 2024-07-27T14:39:40.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:39:40.851 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-27T14:39:40.925 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:40.926 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:41.379 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-07-27T14:39:41.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-07-27T14:39:41.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-07-27T14:39:41.476 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-07-27T14:39:41.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-07-27T14:39:41.573 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:41.573 INFO:tasks.ceph:role: osd.6 2024-07-27T14:39:41.573 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:39:41.573 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-27T14:39:41.646 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:42.147 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-07-27T14:39:42.150 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-07-27T14:39:42.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-07-27T14:39:42.302 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-07-27T14:39:42.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-07-27T14:39:42.401 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-27T14:39:42.402 INFO:tasks.ceph:role: osd.7 2024-07-27T14:39:42.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T14:39:42.402 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-27T14:39:42.476 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-27T14:39:42.971 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-07-27T14:39:42.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-07-27T14:39:42.973 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-07-27T14:39:43.121 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-07-27T14:39:43.154 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:43.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.282+0000 7f3a96585640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-07-27T14:39:43.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.282+0000 7f3a96585640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-07-27T14:39:43.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.283+0000 7f3a96585640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-27T14:39:43.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.283+0000 7f3a96585640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-27T14:39:43.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.283+0000 7f3a96585640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-27T14:39:43.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:43.283+0000 7f3a96585640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-07-27T14:39:46.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-07-27T14:39:46.160 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:46.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.251+0000 7f8ba631c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-07-27T14:39:46.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.252+0000 7f8ba631c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-07-27T14:39:46.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.252+0000 7f8ba631c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-27T14:39:46.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.252+0000 7f8ba631c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-27T14:39:46.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.252+0000 7f8ba631c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-27T14:39:46.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:46.252+0000 7f8ba631c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-07-27T14:39:49.100 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-07-27T14:39:49.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.219+0000 7fc79b01e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.219+0000 7fc79b01e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.219+0000 7fc79b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.219+0000 7fc79b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.219+0000 7fc79b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-27T14:39:49.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:49.220+0000 7fc79b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-07-27T14:39:52.075 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-07-27T14:39:52.104 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:52.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.196+0000 7ffb27cd9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-07-27T14:39:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.196+0000 7ffb27cd9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-07-27T14:39:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.196+0000 7ffb27cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-27T14:39:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.196+0000 7ffb27cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-27T14:39:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.196+0000 7ffb27cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-27T14:39:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:39:52.197+0000 7ffb27cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-07-27T14:39:55.041 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-07-27T14:39:55.071 INFO:tasks.ceph:Reading keys from all nodes... 2024-07-27T14:39:55.071 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-07-27T14:39:55.100 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.101 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-07-27T14:39:55.170 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-07-27T14:39:55.238 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-07-27T14:39:55.306 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-07-27T14:39:55.373 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-07-27T14:39:55.403 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-07-27T14:39:55.473 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.473 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-07-27T14:39:55.541 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.541 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-07-27T14:39:55.608 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-07-27T14:39:55.674 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:39:55.674 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-07-27T14:39:55.693 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:39:55.693 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-07-27T14:39:55.730 INFO:tasks.ceph:Adding keys to all mons... 2024-07-27T14:39:55.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-27T14:39:55.735 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.y] 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCcBqVmmbCSLBAAI80MJOmEOzjD9zr+UWZNDA== 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCgBqVmKdrLDBAABraemv8KDKkFsTJU70IyjA== 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCjBqVmMmebChAANW0mwaVyQu8UIMC3Zb4ouA== 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCmBqVmze7qCBAA+sgSSFXyFzEzNm1vtoSk5g== 2024-07-27T14:39:55.773 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.3] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCpBqVmvL0QBxAAyaYwOWMSdOvbII9OjD6wqw== 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCcBqVmxM4nMRAAMDbyOBRWJOyTKObuEnfH8w== 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.4] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCvBqVme+jkEBAAi4u6X3W88Sp4CWZl/ZMb/g== 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.5] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCyBqVmbK4RDxAAvlCA0dug/LmJkfYErPYbkg== 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.6] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQC1BqVmzpEjDRAAdnogmPqELu6HVe/9DfaGkw== 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.7] 2024-07-27T14:39:55.774 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQC4BqVmik3DCxAA1vyLdSN7bZ5Ja3mVTSNT9g== 2024-07-27T14:39:55.775 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-07-27T14:39:55.775 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCcBqVmqNfGMxAA0D8hLXP9GSOZPjy56a5Wqg== 2024-07-27T14:39:55.775 INFO:teuthology.orchestra.run.smithi087.stdout:[client.1] 2024-07-27T14:39:55.775 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCcBqVmRoX1NhAAti25HJS9uidfiFZgXIQ3gw== 2024-07-27T14:39:55.793 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.y] 2024-07-27T14:39:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCcBqVmmbCSLBAAI80MJOmEOzjD9zr+UWZNDA== 2024-07-27T14:39:55.793 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-07-27T14:39:55.793 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCgBqVmKdrLDBAABraemv8KDKkFsTJU70IyjA== 2024-07-27T14:39:55.793 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCjBqVmMmebChAANW0mwaVyQu8UIMC3Zb4ouA== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCmBqVmze7qCBAA+sgSSFXyFzEzNm1vtoSk5g== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCpBqVmvL0QBxAAyaYwOWMSdOvbII9OjD6wqw== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCcBqVmxM4nMRAAMDbyOBRWJOyTKObuEnfH8w== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCvBqVme+jkEBAAi4u6X3W88Sp4CWZl/ZMb/g== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCyBqVmbK4RDxAAvlCA0dug/LmJkfYErPYbkg== 2024-07-27T14:39:55.794 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.6] 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC1BqVmzpEjDRAAdnogmPqELu6HVe/9DfaGkw== 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.7] 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC4BqVmik3DCxAA1vyLdSN7bZ5Ja3mVTSNT9g== 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCcBqVmqNfGMxAA0D8hLXP9GSOZPjy56a5Wqg== 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2024-07-27T14:39:55.795 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCcBqVmRoX1NhAAti25HJS9uidfiFZgXIQ3gw== 2024-07-27T14:39:55.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-27T14:39:55.818 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-27T14:39:55.887 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:55.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:55.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:55.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:55.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:55.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.051 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.101 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-27T14:39:56.103 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-27T14:39:56.155 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.207 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.209 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.262 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-27T14:39:56.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-27T14:39:56.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-27T14:39:56.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-27T14:39:56.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-27T14:39:56.477 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-07-27T14:39:56.477 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-07-27T14:39:56.501 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-27T14:39:56.597 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-07-27T14:39:56.622 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-07-27T14:39:56.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-27T14:39:56.781 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-07-27T14:39:56.806 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-07-27T14:39:56.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-27T14:39:56.926 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-07-27T14:39:56.951 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:56.953 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-27T14:39:57.009 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-07-27T14:39:57.009 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-27T14:39:57.009 INFO:tasks.ceph.mon.a:Restarting daemon 2024-07-27T14:39:57.009 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-07-27T14:39:57.012 INFO:tasks.ceph.mon.a:Started 2024-07-27T14:39:57.012 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-27T14:39:57.012 INFO:tasks.ceph.mon.c:Restarting daemon 2024-07-27T14:39:57.012 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-07-27T14:39:57.015 INFO:tasks.ceph.mon.c:Started 2024-07-27T14:39:57.015 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-27T14:39:57.016 INFO:tasks.ceph.mon.b:Restarting daemon 2024-07-27T14:39:57.016 DEBUG:teuthology.orchestra.run.smithi112:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-07-27T14:39:57.052 INFO:tasks.ceph.mon.b:Started 2024-07-27T14:39:57.052 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-07-27T14:39:57.052 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-07-27T14:39:57.052 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-07-27T14:39:57.055 INFO:tasks.ceph.mgr.y:Started 2024-07-27T14:39:57.055 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-07-27T14:39:57.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-07-27T14:39:57.057 INFO:tasks.ceph.mgr.x:Started 2024-07-27T14:39:57.057 INFO:tasks.ceph:Setting crush tunables to default 2024-07-27T14:39:57.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2024-07-27T14:39:57.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.156+0000 7f5492d14640 10 monclient: get_monmap_and_config 2024-07-27T14:39:57.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.156+0000 7f5492d14640 10 monclient: build_initial_monmap 2024-07-27T14:39:57.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.156+0000 7f5492d14640 10 monclient: monmap: 2024-07-27T14:39:57.156 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:57.157263+0000 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:57.157263+0000 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:39:57.157 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.156+0000 7f5492d14640 1 Processor -- start 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 -- start start 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient: init 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient: _reopen_session rank -1 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-07-27T14:39:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-c con 0x7f548c0b03b0 addr v2:172.21.15.112:3300/0 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-b con 0x7f548c0b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-a con 0x7f548c140670 addr v2:172.21.15.87:3300/0 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b0790 con 0x7f548c140670 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b4730 con 0x7f548c0b03b0 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f548c0b5380 con 0x7f548c0b58a0 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): _renew_subs 2024-07-27T14:39:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5492d14640 10 monclient(hunting): authenticate will time out at 1137.183686s 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f54912b1640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 msgr2=0x7f548c0b2310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f548bfff640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 msgr2=0x7f548c0b1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f54912b1640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f548c140130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:39:57.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.157+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:39:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f548bfff640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 msgr2=0x7f548c0b1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:39:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f54912b1640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 msgr2=0x7f548c0b2310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:39:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f54912b1640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:39:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:39:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f548c140130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:39:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.358+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:39:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f548bfff640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 msgr2=0x7f548c0b1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:39:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f54912b1640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 msgr2=0x7f548c0b2310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:39:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f548c140130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:39:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:39:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f54912b1640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:39:57.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:57.759+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:39:58.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f548bfff640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 msgr2=0x7f548c0b1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:39:58.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f54912b1640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 msgr2=0x7f548c0b2310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:39:58.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f54912b1640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:39:58.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:39:58.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f548c140130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:39:58.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:39:58.560+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:00.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:00.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:00.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 msgr2=0x7f548c0b1dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f548c0b58a0 0x7f548c0b1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f548c140130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f548c140130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 msgr2=0x7f548c0b2310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f548c140670 0x7f548c0b2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient: _add_conns ranks=[0,1] 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54700036b0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5470007360 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f548c0afb90 con 0x7f54700036b0 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547000c0e0 con 0x7f5470007360 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:00.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f5490ab0640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7f54700036b0 legacy=0x7f5470005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-07-27T14:40:00.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.157+0000 7f548bfff640 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5470007360 legacy=0x7f5470009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-07-27T14:40:00.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.358+0000 7f5490ab0640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7f54700036b0 legacy=0x7f5470005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-07-27T14:40:00.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.358+0000 7f548bfff640 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5470007360 legacy=0x7f5470009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-07-27T14:40:00.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.759+0000 7f5490ab0640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7f54700036b0 legacy=0x7f5470005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-07-27T14:40:00.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:00.759+0000 7f548bfff640 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5470007360 legacy=0x7f5470009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-07-27T14:40:01.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:01.560+0000 7f5490ab0640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7f54700036b0 legacy=0x7f5470005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-07-27T14:40:01.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:01.560+0000 7f548bfff640 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5470007360 legacy=0x7f5470009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5470007360 legacy=0x7f5470009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7f54700036b0 legacy=0x7f5470005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-07-27T14:40:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f54912b1640 1 -- reap_dead start 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.157+0000 7f548a7fc640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f548c0b03b0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f547000c850 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f5470009820 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b0790 con 0x7f547000c850 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b4730 con 0x7f548c0b03b0 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:03.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f548c0b5380 con 0x7f5470009820 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548bfff640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 msgr2=0x7f547000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f54912b1640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f548bfff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f54912b1640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f547000c310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:40:03.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.158+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:03.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f54912b1640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:40:03.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f548bfff640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 msgr2=0x7f547000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:40:03.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f54912b1640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:03.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f548bfff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:03.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f547000c310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:40:03.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.359+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:03.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f54912b1640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:40:03.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f548bfff640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 msgr2=0x7f547000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:40:03.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f54912b1640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:03.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f548bfff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:03.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f547000c310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:40:03.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:03.760+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:04.013 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:00:06.865 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:04.325 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:00:07.178 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:04.344 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:00:07.211 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:04.344 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:00:07.211 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:04.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f548bfff640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 msgr2=0x7f547000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3300/0 2024-07-27T14:40:04.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f54912b1640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.87:3301/0 2024-07-27T14:40:04.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f54912b1640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:04.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f548bfff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:04.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f5490ab0640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f547000c310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:3300/0 2024-07-27T14:40:04.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:04.561+0000 7f5490ab0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:04.670 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:00:07.536 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:04.670 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:00:07.536 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:40:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 -- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 -- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 msgr2=0x7f547000c310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f548c0b03b0 0x7f547000c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 -- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 msgr2=0x7f547000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f547000c850 0x7f547000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient: _add_conns ranks=[1,0] 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f547000b090 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f547000bc90 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547000d320 con 0x7f547000bc90 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f548c0afb90 con 0x7f547000b090 2024-07-27T14:40:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.158+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:06.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.192+0000 7f5490ab0640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f547000b090 0x7f547000b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:55714/0 (socket says 172.21.15.87:55714) 2024-07-27T14:40:06.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.192+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 learned_addr learned my addr 172.21.15.87:0/340147697 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:40:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.753+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f547000bc90 legacy=0x7f5470006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T14:40:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.753+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f547000bc90 legacy=0x7f5470006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:06.753+0000 7f548bfff640 1 --1- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f547000bc90 0x7f5470006020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.112:6789/0 conn(0x7f547000b090 legacy=0x7f547000b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f547000bc90 legacy=0x7f5470006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:09.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 reap_dead start 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54700069d0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f547000cc50 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5470009820 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547000c0e0 con 0x7f54700069d0 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5470006f60 con 0x7f547000cc50 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b0790 con 0x7f5470009820 2024-07-27T14:40:09.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.159+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:09.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.170+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:09.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.170+0000 7f54912b1640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:09.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.170+0000 7f54912b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:09.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.170+0000 7f54912b1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:09.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.170+0000 7f54912b1640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:09.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.171+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T14:40:09.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.171+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:09.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.171+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:09.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.171+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:09.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.373+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.373+0000 7f54912b1640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.373+0000 7f54912b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.373+0000 7f54912b1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.373+0000 7f54912b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.374+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.374+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:09.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.374+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:09.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.374+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:09.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T14:40:09.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:09.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:09.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:09.776+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.578+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.578+0000 7f54912b1640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.578+0000 7f54912b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.578+0000 7f54912b1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.578+0000 7f54912b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.579+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T14:40:10.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.579+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:10.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.579+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:10.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:10.579+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.112:6789/0 conn(0x7f547000cc50 legacy=0x7f5470006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:12.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f54700069d0 legacy=0x7f54700166a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 10 monclient: _add_conns ranks=[2,3] 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.158+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f547000b4a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f547000fb60 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b0790 con 0x7f547000fb60 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f548c0b4730 con 0x7f547000b4a0 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:12.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.159+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 10 monclient(hunting): get_auth_request con 0x7f547000b4a0 auth_method 0 2024-07-27T14:40:12.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:12.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.160+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f547000fb60 auth_method 0 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:12.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T14:40:12.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.161+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T14:40:12.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.361+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 10 monclient(hunting): get_auth_request con 0x7f547000b4a0 auth_method 0 2024-07-27T14:40:12.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:12.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f547000fb60 auth_method 0 2024-07-27T14:40:12.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f548bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.362+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.363+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.363+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.363+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.363+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.363+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f5490ab0640 10 monclient(hunting): get_auth_request con 0x7f547000b4a0 auth_method 0 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f5490ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f5490ab0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.764+0000 7f5490ab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f547000fb60 auth_method 0 2024-07-27T14:40:12.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:12.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:12.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:12.765+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f5490ab0640 10 monclient(hunting): get_auth_request con 0x7f547000b4a0 auth_method 0 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f5490ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f5490ab0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f5490ab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f547000fb60 auth_method 0 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.567+0000 7f548bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:40:13.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-27T14:40:13.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f5490ab0640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:13.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:13.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T14:40:13.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T14:40:13.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:40:13.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f5490ab0640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:13.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:13.568+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T14:40:15.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): tick 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): _check_auth_tickets 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 monclient(hunting): continuing hunt 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 msgr2=0x7f547000f620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000b4a0 0x7f547000f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 msgr2=0x7f5470011f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54700069d0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f54912b1640 1 -- 172.21.15.87:0/340147697 reap_dead start 2024-07-27T14:40:15.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f547000fb60 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5470009820 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547000c310 con 0x7f54700069d0 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b0790 con 0x7f547000fb60 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 1 -- 172.21.15.87:0/340147697 --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0b4730 con 0x7f5470009820 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548a7fc640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:15.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.159+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.87:3300/0 conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:15.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f54912b1640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:15.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f547000fb60 auth_method 0 2024-07-27T14:40:15.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:15.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:15.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f54912b1640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f54912b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f54912b1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.160+0000 7f54912b1640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f54912b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f54912b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:15.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.161+0000 7f54912b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 msgr2=0x7f5470005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 >> v1:172.21.15.87:6789/0 conn(0x7f54700069d0 legacy=0x7f54700144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 10 monclient: _finish_hunting 0 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 1 monclient: found mon.noname-a 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.163+0000 7f548bfff640 1 -- 172.21.15.87:0/340147697 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547000cc50 con 0x7f547000fb60 2024-07-27T14:40:15.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.167+0000 7f548bfff640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f547000fb60 0x7f5470011f80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5470007790 tx=0x7f5478005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1760faf962b10a2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 1 -- 172.21.15.87:0/340147697 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478033070 con 0x7f547000fb60 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: dump: 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.175 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: _finish_auth 0 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: _check_auth_tickets 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 1 -- 172.21.15.87:0/340147697 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f547802d7f0 con 0x7f547000fb60 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 5 monclient: authenticate success, global_id 4107 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:15.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 4 set_mon_vals no callback set 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 1 -- 172.21.15.87:0/340147697 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547802dbe0 con 0x7f547000fb60 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f548b7fe640 10 monclient: dump: 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:15.177 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 10 monclient: shutdown 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 -- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f547000fb60 msgr2=0x7f5470011f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f547000fb60 0x7f5470011f80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5470007790 tx=0x7f5478005040 comp rx=0 tx=0).stop 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 -- 172.21.15.87:0/340147697 shutdown_connections 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 --2- 172.21.15.87:0/340147697 >> v2:172.21.15.112:3300/0 conn(0x7f5470009820 0x7f5470005aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 --2- 172.21.15.87:0/340147697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f547000fb60 0x7f5470011f80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 -- 172.21.15.87:0/340147697 >> 172.21.15.87:0/340147697 conn(0x7f548c073190 msgr2=0x7f548c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 -- 172.21.15.87:0/340147697 shutdown_connections 2024-07-27T14:40:15.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.174+0000 7f5492d14640 1 -- 172.21.15.87:0/340147697 wait complete. 2024-07-27T14:40:15.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.175+0000 7f5492d14640 10 monclient: build_initial_monmap 2024-07-27T14:40:15.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.175+0000 7f5492d14640 10 monclient: monmap: 2024-07-27T14:40:15.181 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:40:15.181 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.175+0000 7f5492d14640 1 librados: starting msgr at 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.175+0000 7f5492d14640 1 librados: starting objecter 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 Processor -- start 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 -- start start 2024-07-27T14:40:15.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 librados: setting wanted keys 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 librados: calling monclient init 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient: init 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54700069d0 0x7f548c144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-b con 0x7f54700069d0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-a con 0x7f5470009820 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000fb60 0x7f548c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): picked mon.noname-c con 0x7f547000fb60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f548c0b4f30 con 0x7f5470009820 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:15.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f548c0b50b0 con 0x7f54700069d0 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f548c140130 con 0x7f547000fb60 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.176+0000 7f5492d14640 10 monclient(hunting): authenticate will time out at 1155.202939s 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f5490ab0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54700069d0 0x7f548c144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57474/0 (socket says 172.21.15.87:57474) 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 1 -- 172.21.15.87:0/3451240360 learned_addr learned my addr 172.21.15.87:0/3451240360 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 10 monclient(hunting): get_auth_request con 0x7f5470009820 auth_method 0 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.177+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): get_auth_request con 0x7f54700069d0 auth_method 0 2024-07-27T14:40:15.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f548bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.178+0000 7f5490ab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 10 monclient(hunting): handle_auth_done global_id 4113 payload 824 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 -- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000fb60 msgr2=0x7f548c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000fb60 0x7f548c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 -- 172.21.15.87:0/3451240360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54700069d0 msgr2=0x7f548c144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54700069d0 0x7f548c144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 10 monclient: _finish_hunting 0 2024-07-27T14:40:15.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 monclient: found mon.noname-a 2024-07-27T14:40:15.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:15.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.181+0000 7f548bfff640 1 -- 172.21.15.87:0/3451240360 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548c146ba0 con 0x7f5470009820 2024-07-27T14:40:15.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.183+0000 7f548bfff640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5478005080 tx=0x7f5478009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.183+0000 7f54897fa640 10 client.?.objecter ms_handle_connect 0x7f5470009820 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.183+0000 7f54897fa640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478046070 con 0x7f5470009820 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: dump: 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:15.188 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: _finish_auth 0 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: _check_auth_tickets 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5478042640 con 0x7f5470009820 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 5 monclient: authenticate success, global_id 4113 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 10 monclient: _renew_subs 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548c146e80 con 0x7f5470009820 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54780429f0 con 0x7f5470009820 2024-07-27T14:40:15.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f54897fa640 10 monclient: dump: 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:15.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f548affd640 4 set_mon_vals no callback set 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 10 monclient: _renew_subs 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f548c0af130 con 0x7f5470009820 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.186+0000 7f5492d14640 1 librados: init done 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.187+0000 7f5492d14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.187+0000 7f5492d14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.187+0000 7f5492d14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.187+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f548c0b1dd0 con 0x7f5470009820 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5478049070 con 0x7f5470009820 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f547804b760 con 0x7f5470009820 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 10 client.4113.objecter ms_dispatch 0x7f548c000b90 osd_map(1..1 src has 1..1) 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-07-27T14:40:15.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.191+0000 7f54897fa640 20 client.4113.objecter dump_active .. 0 homeless 2024-07-27T14:40:15.229 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:15.229+0000 7f7e10a15040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-07-27T14:40:15.301 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:15.301+0000 7fc193f4f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-07-27T14:40:15.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.356+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f5478042b90 con 0x7f5470009820 2024-07-27T14:40:15.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.356+0000 7f54897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:15.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.356+0000 7f54897fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:15.400 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:15.400+0000 7f7e10a15040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-07-27T14:40:15.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.421+0000 7f5492d14640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-27T14:40:15.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.421+0000 7f5492d14640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-27T14:40:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.421+0000 7f5492d14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:40:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.421+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f548c075a50 con 0x7f5470009820 2024-07-27T14:40:15.453 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:15.452+0000 7f7e10a15040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-07-27T14:40:15.466 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:15.466+0000 7fc193f4f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-07-27T14:40:15.526 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:15.526+0000 7fc193f4f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-07-27T14:40:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.547+0000 7f54897fa640 1 -- 172.21.15.87:0/3451240360 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f5478042d70 con 0x7f5470009820 2024-07-27T14:40:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.547+0000 7f54897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-27T14:40:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.547+0000 7f54897fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-07-27T14:40:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.547+0000 7f5492d14640 10 librados: watch_flush enter 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.547+0000 7f5492d14640 10 librados: watch_flush exit 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 20 client.4113.objecter shutdown clearing up homeless session... 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 10 client.4113.objecter successfully canceled tick 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 10 monclient: shutdown 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 msgr2=0x7f548c1458c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5478005080 tx=0x7f5478009630 comp rx=0 tx=0).stop 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 shutdown_connections 2024-07-27T14:40:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f547000fb60 0x7f548c0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5470009820 0x7f548c1458c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 --2- 172.21.15.87:0/3451240360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54700069d0 0x7f548c144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 >> 172.21.15.87:0/3451240360 conn(0x7f548c073190 msgr2=0x7f548c0b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 shutdown_connections 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 -- 172.21.15.87:0/3451240360 wait complete. 2024-07-27T14:40:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:15.548+0000 7f5492d14640 1 librados: shutdown 2024-07-27T14:40:15.560 INFO:tasks.ceph:check_enable_crimson: False 2024-07-27T14:40:15.561 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-07-27T14:40:15.561 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:40:15.561 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-07-27T14:40:15.587 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:40:15.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-07-27T14:40:15.657 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:40:15.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-07-27T14:40:15.707 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:15.706+0000 7f7e10a15040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-07-27T14:40:15.724 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:40:15.724 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-07-27T14:40:15.760 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:15.759+0000 7f7e10a15040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-07-27T14:40:15.790 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:40:15.790 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-07-27T14:40:15.815 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:40:15.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-07-27T14:40:15.841 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:15.841+0000 7fc193f4f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-07-27T14:40:15.880 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:40:15.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-07-27T14:40:15.885 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:15.885+0000 7fc193f4f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-07-27T14:40:15.944 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-27T14:40:15.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-07-27T14:40:16.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new 9f32f839-eba6-44b8-aee5-21988b13d1f9 0 2024-07-27T14:40:16.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.148+0000 7f2de896c640 10 monclient: get_monmap_and_config 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.148+0000 7f2de896c640 10 monclient: build_initial_monmap 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.148+0000 7f2de896c640 10 monclient: monmap: 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:16.149557+0000 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:16.149557+0000 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:16.149 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:16.150 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:16.150 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:16.150 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 Processor -- start 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 -- start start 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient: init 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2de40ab6c0 0x7f2de40ad550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-c con 0x7f2de40ab6c0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de40ada90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40aeb80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2de40abd90 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40ace50 con 0x7f2de40abd90 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f2de40abaa0 con 0x7f2de40ab6c0 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2de40aabc0 con 0x7f2de40aeb80 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.149+0000 7f2de896c640 10 monclient(hunting): authenticate will time out at 1157.178970s 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2de40ab6c0 0x7f2de40ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de1d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de40ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2de40ab6c0 0x7f2de40ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39126/0 (socket says 172.21.15.112:39126) 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 1 -- 172.21.15.112:0/451165491 learned_addr learned my addr 172.21.15.112:0/451165491 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 10 monclient(hunting): get_auth_request con 0x7f2de40ab6c0 auth_method 0 2024-07-27T14:40:16.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.150+0000 7f2de259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): get_auth_request con 0x7f2de40aeb80 auth_method 0 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.151+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-07-27T14:40:16.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 1 -- 172.21.15.112:0/451165491 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 msgr2=0x7f2de40ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:16.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 1 --2- 172.21.15.112:0/451165491 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de40ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 1 -- 172.21.15.112:0/451165491 >> v1:172.21.15.112:6789/0 conn(0x7f2de40abd90 legacy=0x7f2de4141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:16.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 10 monclient: _finish_hunting 0 2024-07-27T14:40:16.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 1 monclient: found mon.noname-c 2024-07-27T14:40:16.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:16.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.153+0000 7f2de259c640 1 -- 172.21.15.112:0/451165491 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40ad130 con 0x7f2de40ab6c0 2024-07-27T14:40:16.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.154+0000 7f2de259c640 1 --2- 172.21.15.112:0/451165491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40ab6c0 0x7f2de40ad550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400aa10 tx=0x7f2dd402ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9252ed103b4a59e8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:16.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 1 -- 172.21.15.112:0/451165491 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd403a070 con 0x7f2de40ab6c0 2024-07-27T14:40:16.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:16.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: dump: 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: _finish_auth 0 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: _check_auth_tickets 2024-07-27T14:40:16.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 1 -- 172.21.15.112:0/451165491 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd402dc10 con 0x7f2de40ab6c0 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de896c640 5 monclient: authenticate success, global_id 4114 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de896c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de896c640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de896c640 4 set_mon_vals no callback set 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 1 -- 172.21.15.112:0/451165491 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400b040 con 0x7f2de40ab6c0 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.161+0000 7f2de159a640 10 monclient: dump: 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.163 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.164 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 10 monclient: shutdown 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 -- 172.21.15.112:0/451165491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40ab6c0 msgr2=0x7f2de40ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 --2- 172.21.15.112:0/451165491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40ab6c0 0x7f2de40ad550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400aa10 tx=0x7f2dd402ce60 comp rx=0 tx=0).stop 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 -- 172.21.15.112:0/451165491 shutdown_connections 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 --2- 172.21.15.112:0/451165491 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de40ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 --2- 172.21.15.112:0/451165491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40ab6c0 0x7f2de40ad550 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 -- 172.21.15.112:0/451165491 >> 172.21.15.112:0/451165491 conn(0x7f2de4067f60 msgr2=0x7f2de40b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 -- 172.21.15.112:0/451165491 shutdown_connections 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.162+0000 7f2de896c640 1 -- 172.21.15.112:0/451165491 wait complete. 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 10 monclient: build_initial_monmap 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 10 monclient: monmap: 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:16.168 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 librados: starting msgr at 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 librados: starting objecter 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 Processor -- start 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 -- start start 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 librados: setting wanted keys 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 1 librados: calling monclient init 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.163+0000 7f2de896c640 10 monclient: init 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2de40ab6c0 0x7f2de41208f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-a con 0x7f2de40ab6c0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40abd90 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de41d88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): picked mon.noname-c con 0x7f2de40aeb80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4142270 con 0x7f2de40ab6c0 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40af090 con 0x7f2de40abd90 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2de41212e0 con 0x7f2de40aeb80 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de896c640 10 monclient(hunting): authenticate will time out at 1157.193457s 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.164+0000 7f2de1d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de2d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de41d88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39132/0 (socket says 172.21.15.112:39132) 2024-07-27T14:40:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2de40ab6c0 0x7f2de41208f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 1 -- 172.21.15.112:0/1410259892 learned_addr learned my addr 172.21.15.112:0/1410259892 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): get_auth_request con 0x7f2de40abd90 auth_method 0 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de2d9d640 10 monclient(hunting): get_auth_request con 0x7f2de40aeb80 auth_method 0 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de2d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de259c640 10 monclient(hunting): get_auth_request con 0x7f2de40ab6c0 auth_method 0 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de259c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.165+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.166+0000 7f2de259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 -- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 msgr2=0x7f2de41d88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de41d88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 -- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2de40ab6c0 msgr2=0x7f2de41208f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2de40ab6c0 0x7f2de41208f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 10 monclient: _finish_hunting 0 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 monclient: found mon.noname-b 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.168+0000 7f2de1d9b640 1 -- 172.21.15.112:0/1410259892 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de4122460 con 0x7f2de40abd90 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.170+0000 7f2de1d9b640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e870 tx=0x7f2dd800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.170+0000 7f2dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2de40abd90 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.170+0000 7f2dd2ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8017070 con 0x7f2de40abd90 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 10 monclient: dump: 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 10 monclient: _finish_auth 0 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2de896c640 5 monclient: authenticate success, global_id 4117 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2de896c640 10 monclient: _renew_subs 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2de896c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.172+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de41227a0 con 0x7f2de40abd90 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 monclient: _renew_subs 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de41d8de0 con 0x7f2de40abd90 2024-07-27T14:40:16.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 1 librados: init done 2024-07-27T14:40:16.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.173+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de40af4f0 con 0x7f2de40abd90 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd8005990 con 0x7f2de40abd90 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8021060 con 0x7f2de40abd90 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: dump: 2024-07-27T14:40:16.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2dd801a070 con 0x7f2de40abd90 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2de0d99640 4 set_mon_vals no callback set 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f2dd8006750 con 0x7f2de40abd90 2024-07-27T14:40:16.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 client.4117.objecter ms_dispatch 0x7f2de4000b90 osd_map(2..2 src has 1..2) 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f2de41227a0 con 0x7f2de40abd90 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.369+0000 7f2dd2ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:16.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.467+0000 7f2de896c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9f32f839-eba6-44b8-aee5-21988b13d1f9", "id": 0}] 2024-07-27T14:40:16.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.467+0000 7f2de896c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9f32f839-eba6-44b8-aee5-21988b13d1f9", "id": 0}] 2024-07-27T14:40:16.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.467+0000 7f2de896c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.467+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9f32f839-eba6-44b8-aee5-21988b13d1f9", "id": 0} v 0) -- 0x7f2de40b2910 con 0x7f2de40abd90 2024-07-27T14:40:16.613 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:16.612+0000 7f7e10a15040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-07-27T14:40:16.664 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:16.664+0000 7fc193f4f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-07-27T14:40:16.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.695+0000 7f2dd2ffd640 1 -- 172.21.15.112:0/1410259892 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f32f839-eba6-44b8-aee5-21988b13d1f9", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f2dd800c940 con 0x7f2de40abd90 2024-07-27T14:40:16.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.695+0000 7f2dd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9f32f839-eba6-44b8-aee5-21988b13d1f9", "id": 0}] 2024-07-27T14:40:16.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.695+0000 7f2dd2ffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:16.696 INFO:teuthology.orchestra.run.smithi112.stdout:0 2024-07-27T14:40:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 10 librados: watch_flush enter 2024-07-27T14:40:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 10 librados: watch_flush exit 2024-07-27T14:40:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 20 client.4117.objecter shutdown clearing up homeless session... 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 10 client.4117.objecter successfully canceled tick 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 10 monclient: shutdown 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 msgr2=0x7f2de41d8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e870 tx=0x7f2dd800ed30 comp rx=0 tx=0).stop 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 shutdown_connections 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2de40aeb80 0x7f2de41d88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2de40abd90 0x7f2de41d8360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 --2- 172.21.15.112:0/1410259892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2de40ab6c0 0x7f2de41208f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 >> 172.21.15.112:0/1410259892 conn(0x7f2de4067f60 msgr2=0x7f2de41423f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 shutdown_connections 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 -- 172.21.15.112:0/1410259892 wait complete. 2024-07-27T14:40:16.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.696+0000 7f2de896c640 1 librados: shutdown 2024-07-27T14:40:16.706 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:16.706+0000 7f7e10a15040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-07-27T14:40:16.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new ddadaf3f-715f-4ea5-80de-b3266c50cbf4 1 2024-07-27T14:40:16.755 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:16.754+0000 7f7e10a15040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-07-27T14:40:16.762 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:16.762+0000 7fc193f4f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-07-27T14:40:16.806 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:16.805+0000 7fc193f4f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.814+0000 7f9d1a805640 10 monclient: get_monmap_and_config 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.814+0000 7f9d1a805640 10 monclient: build_initial_monmap 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.814+0000 7f9d1a805640 10 monclient: monmap: 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:16.815060+0000 2024-07-27T14:40:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:16.815060+0000 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:16.815 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 Processor -- start 2024-07-27T14:40:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 -- start start 2024-07-27T14:40:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient: init 2024-07-27T14:40:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d140b03b0 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 0x7f9d140b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-b con 0x7f9d140b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 0x7f9d1414d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-c con 0x7f9d140b2310 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d140b4730 con 0x7f9d140b03b0 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f9d140b0790 con 0x7f9d140b2310 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f9d140b2780 con 0x7f9d140b58a0 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.815+0000 7f9d1a805640 10 monclient(hunting): authenticate will time out at 1157.844954s 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 0x7f9d1414d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 0x7f9d1414d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39144/0 (socket says 172.21.15.112:39144) 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 1 -- 172.21.15.112:0/3683133763 learned_addr learned my addr 172.21.15.112:0/3683133763 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 10 monclient(hunting): get_auth_request con 0x7f9d140b2310 auth_method 0 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.817+0000 7f9d18da2640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d137fe640 1 --2- 172.21.15.112:0/3683133763 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 0x7f9d140b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d137fe640 10 monclient(hunting): get_auth_request con 0x7f9d140b58a0 auth_method 0 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d137fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.818+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.819+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.819+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.819+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.820+0000 7f9d137fe640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.820+0000 7f9d137fe640 1 -- 172.21.15.112:0/3683133763 >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 msgr2=0x7f9d1414d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.820+0000 7f9d137fe640 1 --2- 172.21.15.112:0/3683133763 >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 0x7f9d1414d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 1 -- 172.21.15.112:0/3683133763 >> v1:172.21.15.112:6789/0 conn(0x7f9d140b03b0 legacy=0x7f9d14140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 10 monclient: _finish_hunting 0 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 1 monclient: found mon.noname-b 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:40:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.821+0000 7f9d137fe640 1 -- 172.21.15.112:0/3683133763 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d140b5380 con 0x7f9d140b58a0 2024-07-27T14:40:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.823+0000 7f9d137fe640 1 --2- 172.21.15.112:0/3683133763 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 0x7f9d140b1dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d080097a0 tx=0x7f9d0802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=736570a243f77516 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:16.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 1 -- 172.21.15.112:0/3683133763 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08038070 con 0x7f9d140b58a0 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: dump: 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.828 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: _finish_auth 0 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 1 -- 172.21.15.112:0/3683133763 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d08031700 con 0x7f9d140b58a0 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d1a805640 5 monclient: authenticate success, global_id 4109 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d1a805640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d1a805640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d1a805640 4 set_mon_vals no callback set 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 1 -- 172.21.15.112:0/3683133763 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08031ad0 con 0x7f9d140b58a0 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.827+0000 7f9d12ffd640 10 monclient: dump: 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.830 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 10 monclient: shutdown 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 -- 172.21.15.112:0/3683133763 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 msgr2=0x7f9d140b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 --2- 172.21.15.112:0/3683133763 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 0x7f9d140b1dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d080097a0 tx=0x7f9d0802b7a0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 -- 172.21.15.112:0/3683133763 shutdown_connections 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 --2- 172.21.15.112:0/3683133763 >> v2:172.21.15.112:3300/0 conn(0x7f9d140b2310 0x7f9d1414d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 --2- 172.21.15.112:0/3683133763 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b58a0 0x7f9d140b1dd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 -- 172.21.15.112:0/3683133763 >> 172.21.15.112:0/3683133763 conn(0x7f9d14073190 msgr2=0x7f9d14067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 -- 172.21.15.112:0/3683133763 shutdown_connections 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.828+0000 7f9d1a805640 1 -- 172.21.15.112:0/3683133763 wait complete. 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 10 monclient: build_initial_monmap 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 10 monclient: monmap: 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:16.834 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 librados: starting msgr at 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 librados: starting objecter 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 Processor -- start 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 -- start start 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 librados: setting wanted keys 2024-07-27T14:40:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 1 librados: calling monclient init 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.829+0000 7f9d1a805640 10 monclient: init 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-b con 0x7f9d140b03b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 0x7f9d14136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-c con 0x7f9d140b2310 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d140b58a0 0x7f9d14137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): picked mon.noname-a con 0x7f9d140b58a0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d140b3110 con 0x7f9d140b58a0 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9d140b2f90 con 0x7f9d140b03b0 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f9d140b4f30 con 0x7f9d140b2310 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d1a805640 10 monclient(hunting): authenticate will time out at 1157.859391s 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.830+0000 7f9d137fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 0x7f9d14136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d18da2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d140b58a0 0x7f9d14137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39158/0 (socket says 172.21.15.112:39158) 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 1 -- 172.21.15.112:0/2888237351 learned_addr learned my addr 172.21.15.112:0/2888237351 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): get_auth_request con 0x7f9d140b03b0 auth_method 0 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): get_auth_request con 0x7f9d140b2310 auth_method 0 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d18da2640 10 monclient(hunting): get_auth_request con 0x7f9d140b58a0 auth_method 0 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d18da2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d18da2640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d18da2640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.831+0000 7f9d13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.832+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.832+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.832+0000 7f9d18da2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:16.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-07-27T14:40:16.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 -- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 msgr2=0x7f9d14136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 0x7f9d14136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 -- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d140b58a0 msgr2=0x7f9d14137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d140b58a0 0x7f9d14137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 10 monclient: _finish_hunting 0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 monclient: found mon.noname-b 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d13fff640 1 -- 172.21.15.112:0/2888237351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d14146340 con 0x7f9d140b03b0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.834+0000 7f9d137fe640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 0x7f9d14136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.837+0000 7f9d13fff640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400b3e0 tx=0x7f9d0400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.837+0000 7f9d10ff9640 10 client.?.objecter ms_handle_connect 0x7f9d140b03b0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.837+0000 7f9d10ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04015020 con 0x7f9d140b03b0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 10 monclient: dump: 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 10 monclient: _finish_auth 0 2024-07-27T14:40:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 5 monclient: authenticate success, global_id 4123 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 10 monclient: _renew_subs 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d14146620 con 0x7f9d140b03b0 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 10 monclient: _renew_subs 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d14146a40 con 0x7f9d140b03b0 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.839+0000 7f9d1a805640 1 librados: init done 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.840+0000 7f9d1a805640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.840+0000 7f9d1a805640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.840+0000 7f9d1a805640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.840+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d140b1dd0 con 0x7f9d140b03b0 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d04011830 con 0x7f9d140b03b0 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d040091f0 con 0x7f9d140b03b0 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d10ff9640 10 monclient: dump: 2024-07-27T14:40:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.841+0000 7f9d127fc640 4 set_mon_vals no callback set 2024-07-27T14:40:16.957 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:16.957+0000 7f7e10a15040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9d04018020 con 0x7f9d140b03b0 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f9d0401a7b0 con 0x7f9d140b03b0 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 10 client.4123.objecter ms_dispatch 0x7f9d14000b90 osd_map(3..3 src has 1..3) 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 20 client.4123.objecter dump_active .. 0 homeless 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f9d0401f020 con 0x7f9d140b03b0 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:16.969+0000 7f9d10ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:16.975 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:16.975+0000 7fc193f4f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-07-27T14:40:17.006 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.006+0000 7f7e10a15040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-07-27T14:40:17.027 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.026+0000 7fc193f4f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-07-27T14:40:17.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.042+0000 7f9d1a805640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ddadaf3f-715f-4ea5-80de-b3266c50cbf4", "id": 1}] 2024-07-27T14:40:17.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.042+0000 7f9d1a805640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ddadaf3f-715f-4ea5-80de-b3266c50cbf4", "id": 1}] 2024-07-27T14:40:17.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.042+0000 7f9d1a805640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.042+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ddadaf3f-715f-4ea5-80de-b3266c50cbf4", "id": 1} v 0) -- 0x7f9d1413ee00 con 0x7f9d140b03b0 2024-07-27T14:40:17.061 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.060+0000 7f7e10a15040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-07-27T14:40:17.082 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.082+0000 7fc193f4f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-07-27T14:40:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.147+0000 7f9d10ff9640 1 -- 172.21.15.112:0/2888237351 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ddadaf3f-715f-4ea5-80de-b3266c50cbf4", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f9d040249c0 con 0x7f9d140b03b0 2024-07-27T14:40:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.147+0000 7f9d10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ddadaf3f-715f-4ea5-80de-b3266c50cbf4", "id": 1}] 2024-07-27T14:40:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.147+0000 7f9d10ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:17.148 INFO:teuthology.orchestra.run.smithi112.stdout:1 2024-07-27T14:40:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.147+0000 7f9d1a805640 10 librados: watch_flush enter 2024-07-27T14:40:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 10 librados: watch_flush exit 2024-07-27T14:40:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 20 client.4123.objecter shutdown clearing up homeless session... 2024-07-27T14:40:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 10 client.4123.objecter successfully canceled tick 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 10 monclient: shutdown 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 msgr2=0x7f9d141442e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400b3e0 tx=0x7f9d0400b8a0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 shutdown_connections 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d140b58a0 0x7f9d14137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d140b2310 0x7f9d14136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 --2- 172.21.15.112:0/2888237351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d140b03b0 0x7f9d141442e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 >> 172.21.15.112:0/2888237351 conn(0x7f9d14073190 msgr2=0x7f9d14067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 shutdown_connections 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 -- 172.21.15.112:0/2888237351 wait complete. 2024-07-27T14:40:17.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.148+0000 7f9d1a805640 1 librados: shutdown 2024-07-27T14:40:17.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new 5276af43-fb52-45c0-b5f8-35e0ed1499e2 2 2024-07-27T14:40:17.180 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.179+0000 7f7e10a15040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-07-27T14:40:17.200 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.200+0000 7fc193f4f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-07-27T14:40:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.277+0000 7f2ba235a640 10 monclient: get_monmap_and_config 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.277+0000 7f2ba235a640 10 monclient: build_initial_monmap 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.277+0000 7f2ba235a640 10 monclient: monmap: 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:17.278638+0000 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:17.278638+0000 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:17.280 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.278+0000 7f2ba235a640 1 Processor -- start 2024-07-27T14:40:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.278+0000 7f2ba235a640 1 -- start start 2024-07-27T14:40:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.278+0000 7f2ba235a640 10 monclient: init 2024-07-27T14:40:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0b03b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b9c0b58a0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f2b9c0b24e0 0x7f2b9c14d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c0b24e0 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b9c0b4730 con 0x7f2b9c0b58a0 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f2b9c0b0790 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2b9c0b5380 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba235a640 10 monclient(hunting): authenticate will time out at 1158.308114s 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2b9bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.279+0000 7f2ba08f7640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f2b9c0b24e0 0x7f2b9c14d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.112:54286/0 (socket says 172.21.15.112:54286) 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 1 -- 172.21.15.112:0/1719556924 learned_addr learned my addr 172.21.15.112:0/1719556924 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): get_auth_request con 0x7f2b9c0b03b0 auth_method 0 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2ba08f7640 10 monclient(hunting): get_auth_request con 0x7f2b9c0b24e0 auth_method 0 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2ba08f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2ba08f7640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2ba08f7640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.280+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.281+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.281+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.281+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 1 -- 172.21.15.112:0/1719556924 >> v2:172.21.15.87:3300/0 conn(0x7f2b9c0b24e0 msgr2=0x7f2b9c14d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 1 --2- 172.21.15.112:0/1719556924 >> v2:172.21.15.87:3300/0 conn(0x7f2b9c0b24e0 0x7f2b9c14d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 1 -- 172.21.15.112:0/1719556924 >> v1:172.21.15.87:6789/0 conn(0x7f2b9c0b58a0 legacy=0x7f2b9c0b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 10 monclient: _finish_hunting 0 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 1 monclient: found mon.noname-b 2024-07-27T14:40:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.282+0000 7f2b9bfff640 1 -- 172.21.15.112:0/1719556924 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9c0afb90 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.284+0000 7f2b9bfff640 1 --2- 172.21.15.112:0/1719556924 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009870 tx=0x7f2b8c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60b5ba08f2bbcb66 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 1 -- 172.21.15.112:0/1719556924 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c035020 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: dump: 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: _finish_auth 0 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: _check_auth_tickets 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 1 -- 172.21.15.112:0/1719556924 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8c031640 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 1 -- 172.21.15.112:0/1719556924 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02d040 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.287+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2b9affd640 10 monclient: dump: 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 5 monclient: authenticate success, global_id 4115 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 4 set_mon_vals no callback set 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 10 monclient: shutdown 2024-07-27T14:40:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 -- 172.21.15.112:0/1719556924 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 msgr2=0x7f2b9c140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 --2- 172.21.15.112:0/1719556924 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009870 tx=0x7f2b8c02b8f0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 -- 172.21.15.112:0/1719556924 shutdown_connections 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 --2- 172.21.15.112:0/1719556924 >> v2:172.21.15.87:3300/0 conn(0x7f2b9c0b24e0 0x7f2b9c14d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 --2- 172.21.15.112:0/1719556924 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b03b0 0x7f2b9c140130 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 -- 172.21.15.112:0/1719556924 >> 172.21.15.112:0/1719556924 conn(0x7f2b9c073190 msgr2=0x7f2b9c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 -- 172.21.15.112:0/1719556924 shutdown_connections 2024-07-27T14:40:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.288+0000 7f2ba235a640 1 -- 172.21.15.112:0/1719556924 wait complete. 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 10 monclient: build_initial_monmap 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 10 monclient: monmap: 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 librados: starting msgr at 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 librados: starting objecter 2024-07-27T14:40:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 Processor -- start 2024-07-27T14:40:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 -- start start 2024-07-27T14:40:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 librados: setting wanted keys 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 1 librados: calling monclient init 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.289+0000 7f2ba235a640 10 monclient: init 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b9c0b03b0 0x7f2b9c146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0b03b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9c0b24e0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b9c0b58a0 0x7f2b9c1d8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c0b58a0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c0b4f30 con 0x7f2b9c0b58a0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c0b50b0 con 0x7f2b9c0b03b0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2b9c140670 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.290+0000 7f2ba235a640 10 monclient(hunting): authenticate will time out at 1158.319445s 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b9c0b03b0 0x7f2b9c146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2ba08f7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b9c0b58a0 0x7f2b9c1d8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.112:54294/0 (socket says 172.21.15.112:54294) 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 1 -- 172.21.15.112:0/2106550183 learned_addr learned my addr 172.21.15.112:0/2106550183 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2b9c0b24e0 auth_method 0 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): get_auth_request con 0x7f2b9c0b03b0 auth_method 0 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2ba08f7640 10 monclient(hunting): get_auth_request con 0x7f2b9c0b58a0 auth_method 0 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2ba08f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2ba08f7640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2ba08f7640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.291+0000 7f2b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.292+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.292+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.292+0000 7f2ba08f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 -- 172.21.15.112:0/2106550183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b9c0b03b0 msgr2=0x7f2b9c146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b9c0b03b0 0x7f2b9c146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 -- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b9c0b58a0 msgr2=0x7f2b9c1d8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b9c0b58a0 0x7f2b9c1d8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 monclient: found mon.noname-c 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.294+0000 7f2b9b7fe640 1 -- 172.21.15.112:0/2106550183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9c0abd60 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.296+0000 7f2b9b7fe640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c4a0 tx=0x7f2b9000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.296+0000 7f2b98ff9640 10 client.?.objecter ms_handle_connect 0x7f2b9c0b24e0 2024-07-27T14:40:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.296+0000 7f2b98ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b900052d0 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: dump: 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.311 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.312 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.312 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.312 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.312 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: _finish_auth 0 2024-07-27T14:40:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b90005470 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90007260 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 5 monclient: authenticate success, global_id 4118 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b98ff9640 10 monclient: dump: 2024-07-27T14:40:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 10 monclient: _renew_subs 2024-07-27T14:40:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9c0abfd0 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2b9a7fc640 4 set_mon_vals no callback set 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 10 monclient: _renew_subs 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9c0ab1f0 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.300+0000 7f2ba235a640 1 librados: init done 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.301+0000 7f2ba235a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.301+0000 7f2ba235a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.301+0000 7f2ba235a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.301+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b9c140130 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2b90007400 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f2b90011590 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 10 client.4118.objecter ms_dispatch 0x7f2b9c000b90 osd_map(4..4 src has 1..4) 2024-07-27T14:40:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 3 client.4118.objecter handle_osd_map got epochs [4,4] > 0 2024-07-27T14:40:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 3 client.4118.objecter handle_osd_map decoding full epoch 4 2024-07-27T14:40:17.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.306+0000 7f2b98ff9640 20 client.4118.objecter dump_active .. 0 homeless 2024-07-27T14:40:17.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.478+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f2b9c0abfd0 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.478+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.478+0000 7f2b98ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:17.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.551+0000 7f2ba235a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5276af43-fb52-45c0-b5f8-35e0ed1499e2", "id": 2}] 2024-07-27T14:40:17.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.551+0000 7f2ba235a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5276af43-fb52-45c0-b5f8-35e0ed1499e2", "id": 2}] 2024-07-27T14:40:17.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.551+0000 7f2ba235a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:17.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.551+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5276af43-fb52-45c0-b5f8-35e0ed1499e2", "id": 2} v 0) -- 0x7f2b9c0b7940 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.572 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.571+0000 7f7e10a15040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-07-27T14:40:17.586 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.586+0000 7fc193f4f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-07-27T14:40:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.681+0000 7f2b98ff9640 1 -- 172.21.15.112:0/2106550183 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5276af43-fb52-45c0-b5f8-35e0ed1499e2", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f2b900039b0 con 0x7f2b9c0b24e0 2024-07-27T14:40:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.681+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5276af43-fb52-45c0-b5f8-35e0ed1499e2", "id": 2}] 2024-07-27T14:40:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.681+0000 7f2b98ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:17.682 INFO:teuthology.orchestra.run.smithi112.stdout:2 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 10 librados: watch_flush enter 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 10 librados: watch_flush exit 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 20 client.4118.objecter shutdown clearing up homeless session... 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 10 client.4118.objecter successfully canceled tick 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 10 monclient: shutdown 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 msgr2=0x7f2b9c0aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000c4a0 tx=0x7f2b9000c960 comp rx=0 tx=0).stop 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 shutdown_connections 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b9c0b58a0 0x7f2b9c1d8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b9c0b24e0 0x7f2b9c0aa6d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 --2- 172.21.15.112:0/2106550183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b9c0b03b0 0x7f2b9c146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.682+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 >> 172.21.15.112:0/2106550183 conn(0x7f2b9c073190 msgr2=0x7f2b9c0b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.683+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 shutdown_connections 2024-07-27T14:40:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.683+0000 7f2ba235a640 1 -- 172.21.15.112:0/2106550183 wait complete. 2024-07-27T14:40:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.683+0000 7f2ba235a640 1 librados: shutdown 2024-07-27T14:40:17.694 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new a14626c8-e29d-4fb7-aec6-179ac404e813 3 2024-07-27T14:40:17.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.796+0000 7f7407ba7640 10 monclient: get_monmap_and_config 2024-07-27T14:40:17.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.796+0000 7f7407ba7640 10 monclient: build_initial_monmap 2024-07-27T14:40:17.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.796+0000 7f7407ba7640 10 monclient: monmap: 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:17.797754+0000 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:17.797754+0000 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.797+0000 7f7407ba7640 1 Processor -- start 2024-07-27T14:40:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.797+0000 7f7407ba7640 1 -- start start 2024-07-27T14:40:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.797+0000 7f7407ba7640 10 monclient: init 2024-07-27T14:40:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-07-27T14:40:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f74000b03b0 0x7f7400140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-c con 0x7f74000b03b0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74000b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-b con 0x7f74000b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f7400140670 0x7f74000b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-a con 0x7f7400140670 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f74000b0790 con 0x7f7400140670 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f74000b4730 con 0x7f74000b03b0 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f74000b5380 con 0x7f74000b58a0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7407ba7640 10 monclient(hunting): authenticate will time out at 1158.827223s 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7405943640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f74000b03b0 0x7f7400140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7406144640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f7400140670 0x7f74000b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.798+0000 7f7405142640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74000b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f74000b03b0 0x7f7400140130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39170/0 (socket says 172.21.15.112:39170) 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 1 -- 172.21.15.112:0/36433622 learned_addr learned my addr 172.21.15.112:0/36433622 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): get_auth_request con 0x7f74000b03b0 auth_method 0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): get_auth_request con 0x7f7400140670 auth_method 0 2024-07-27T14:40:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): get_auth_request con 0x7f74000b58a0 auth_method 0 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.799+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 -- 172.21.15.112:0/36433622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 msgr2=0x7f74000b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 --2- 172.21.15.112:0/36433622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74000b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 -- 172.21.15.112:0/36433622 >> v2:172.21.15.87:3300/0 conn(0x7f7400140670 msgr2=0x7f74000b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 --2- 172.21.15.112:0/36433622 >> v2:172.21.15.87:3300/0 conn(0x7f7400140670 0x7f74000b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 10 monclient: _finish_hunting 0 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 monclient: found mon.noname-c 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.801+0000 7f7405943640 1 -- 172.21.15.112:0/36433622 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74000afb90 con 0x7f74000b03b0 2024-07-27T14:40:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.803+0000 7f7405943640 1 --2- 172.21.15.112:0/36433622 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f7400140130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f74000736a0 tx=0x7f73f802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=593d79e4c126cd55 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.806+0000 7f7404941640 1 -- 172.21.15.112:0/36433622 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f802e070 con 0x7f74000b03b0 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: dump: 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: _finish_auth 0 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: _check_auth_tickets 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 1 -- 172.21.15.112:0/36433622 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f8037070 con 0x7f74000b03b0 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 5 monclient: authenticate success, global_id 4129 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 4 set_mon_vals no callback set 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 1 -- 172.21.15.112:0/36433622 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f8031a00 con 0x7f74000b03b0 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7404941640 10 monclient: dump: 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.810 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 10 monclient: shutdown 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 -- 172.21.15.112:0/36433622 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 msgr2=0x7f7400140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 --2- 172.21.15.112:0/36433622 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f7400140130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f74000736a0 tx=0x7f73f802b880 comp rx=0 tx=0).stop 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 -- 172.21.15.112:0/36433622 shutdown_connections 2024-07-27T14:40:17.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 --2- 172.21.15.112:0/36433622 >> v2:172.21.15.87:3300/0 conn(0x7f7400140670 0x7f74000b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 --2- 172.21.15.112:0/36433622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74000b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 --2- 172.21.15.112:0/36433622 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f7400140130 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 -- 172.21.15.112:0/36433622 >> 172.21.15.112:0/36433622 conn(0x7f7400073190 msgr2=0x7f7400067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 -- 172.21.15.112:0/36433622 shutdown_connections 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.807+0000 7f7407ba7640 1 -- 172.21.15.112:0/36433622 wait complete. 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.808+0000 7f7407ba7640 10 monclient: build_initial_monmap 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.808+0000 7f7407ba7640 10 monclient: monmap: 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:17.814 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.808+0000 7f7407ba7640 1 librados: starting msgr at 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.808+0000 7f7407ba7640 1 librados: starting objecter 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 Processor -- start 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 -- start start 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 librados: setting wanted keys 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 librados: calling monclient init 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient: init 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:40:17.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-b con 0x7f74000b03b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74001d8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-c con 0x7f74000b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 0x7f74001d89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): picked mon.noname-a con 0x7f7400140670 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f74000b4f30 con 0x7f7400140670 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74000b50b0 con 0x7f74000b03b0 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f74000b3430 con 0x7f74000b58a0 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.809+0000 7f7407ba7640 10 monclient(hunting): authenticate will time out at 1158.838662s 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7406144640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 0x7f74001d89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405142640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74001d8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39182/0 (socket says 172.21.15.112:39182) 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 1 -- 172.21.15.112:0/4177405038 learned_addr learned my addr 172.21.15.112:0/4177405038 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): get_auth_request con 0x7f74000b03b0 auth_method 0 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7406144640 10 monclient(hunting): get_auth_request con 0x7f7400140670 auth_method 0 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7406144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7406144640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7406144640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405142640 10 monclient(hunting): get_auth_request con 0x7f74000b58a0 auth_method 0 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405142640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405142640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.810+0000 7f7405943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7406144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.811+0000 7f7405142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:17.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 -- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 msgr2=0x7f74001d8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74001d8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 -- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 msgr2=0x7f74001d89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 0x7f74001d89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 10 monclient: _finish_hunting 0 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 monclient: found mon.noname-b 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7405943640 1 -- 172.21.15.112:0/4177405038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74000aef50 con 0x7f74000b03b0 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.813+0000 7f7406144640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 0x7f74001d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.816+0000 7f7405943640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000c00 tx=0x7f73f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.816+0000 7f73fe7fc640 10 client.?.objecter ms_handle_connect 0x7f74000b03b0 2024-07-27T14:40:17.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.816+0000 7f73fe7fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f8032670 con 0x7f74000b03b0 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: dump: 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: _finish_auth 0 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f8032830 con 0x7f74000b03b0 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 5 monclient: authenticate success, global_id 4132 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 10 monclient: _renew_subs 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74000af230 con 0x7f74000b03b0 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f8032c20 con 0x7f74000b03b0 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fe7fc640 10 monclient: dump: 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f73fffff640 4 set_mon_vals no callback set 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 10 monclient: _renew_subs 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74000ad660 con 0x7f74000b03b0 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.822+0000 7f7407ba7640 1 librados: init done 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.823+0000 7f7407ba7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.823+0000 7f7407ba7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.823+0000 7f7407ba7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.823+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7400140130 con 0x7f74000b03b0 2024-07-27T14:40:17.860 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.859+0000 7f7e10a15040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-07-27T14:40:17.865 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.865+0000 7fc193f4f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-07-27T14:40:17.913 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:17.913+0000 7f7e10a15040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-07-27T14:40:17.918 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:17.917+0000 7fc193f4f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f73f80453f0 con 0x7f74000b03b0 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f73f803b880 con 0x7f74000b03b0 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 10 client.4132.objecter ms_dispatch 0x7f7400000b90 osd_map(5..5 src has 1..5) 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f73f8040070 con 0x7f74000b03b0 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.926+0000 7f73fe7fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:17.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.997+0000 7f7407ba7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a14626c8-e29d-4fb7-aec6-179ac404e813", "id": 3}] 2024-07-27T14:40:17.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.997+0000 7f7407ba7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a14626c8-e29d-4fb7-aec6-179ac404e813", "id": 3}] 2024-07-27T14:40:17.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.997+0000 7f7407ba7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:17.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:17.997+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a14626c8-e29d-4fb7-aec6-179ac404e813", "id": 3} v 0) -- 0x7f740006c230 con 0x7f74000b03b0 2024-07-27T14:40:18.034 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.034+0000 7f7e10a15040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-07-27T14:40:18.039 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.038+0000 7fc193f4f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-07-27T14:40:18.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.091+0000 7f73fe7fc640 1 -- 172.21.15.112:0/4177405038 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a14626c8-e29d-4fb7-aec6-179ac404e813", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f73f803b3f0 con 0x7f74000b03b0 2024-07-27T14:40:18.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.091+0000 7f73fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a14626c8-e29d-4fb7-aec6-179ac404e813", "id": 3}] 2024-07-27T14:40:18.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.091+0000 7f73fe7fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:18.092 INFO:teuthology.orchestra.run.smithi112.stdout:3 2024-07-27T14:40:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 10 librados: watch_flush enter 2024-07-27T14:40:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 10 librados: watch_flush exit 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 20 client.4132.objecter shutdown clearing up homeless session... 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 10 client.4132.objecter successfully canceled tick 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 10 monclient: shutdown 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 msgr2=0x7f74000ad120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000c00 tx=0x7f73f80046c0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 shutdown_connections 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7400140670 0x7f74001d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74000b58a0 0x7f74001d8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 --2- 172.21.15.112:0/4177405038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74000b03b0 0x7f74000ad120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 >> 172.21.15.112:0/4177405038 conn(0x7f7400073190 msgr2=0x7f74000b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 shutdown_connections 2024-07-27T14:40:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 -- 172.21.15.112:0/4177405038 wait complete. 2024-07-27T14:40:18.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.092+0000 7f7407ba7640 1 librados: shutdown 2024-07-27T14:40:18.103 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new a73fab50-80f0-4936-a1f4-91b6fd224114 4 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.200+0000 7f2b6d0c5640 10 monclient: get_monmap_and_config 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.200+0000 7f2b6d0c5640 10 monclient: build_initial_monmap 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.200+0000 7f2b6d0c5640 10 monclient: monmap: 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:18.201720+0000 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:18.201720+0000 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:18.202 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:18.202 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:18.202 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:18.202 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 Processor -- start 2024-07-27T14:40:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 -- start start 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient: init 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b68072ff0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f2b680b4650 0x7f2b6813ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2b680b4650 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b680b2df0 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b680ac610 con 0x7f2b68072ff0 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b680ac300 con 0x7f2b680b2df0 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f2b680aab70 con 0x7f2b680b4650 2024-07-27T14:40:18.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.201+0000 7f2b6d0c5640 10 monclient(hunting): authenticate will time out at 1159.230971s 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.202+0000 7f2b6759e640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f2b680b2df0 0x7f2b68140470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46266/0 (socket says 172.21.15.112:46266) 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.202+0000 7f2b6759e640 1 -- 172.21.15.112:0/2563596753 learned_addr learned my addr 172.21.15.112:0/2563596753 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.202+0000 7f2b6659c640 1 --2- 172.21.15.112:0/2563596753 >> v2:172.21.15.87:3300/0 conn(0x7f2b680b4650 0x7f2b6813ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.203+0000 7f2b6659c640 10 monclient(hunting): get_auth_request con 0x7f2b680b4650 auth_method 0 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.203+0000 7f2b6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.203+0000 7f2b6659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.203+0000 7f2b6659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.204+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.204+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.204+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 1 -- 172.21.15.112:0/2563596753 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762232791 0 0) 0x7f2b680ac300 con 0x7f2b680b2df0 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 10 monclient(hunting): _negotiate 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 10 monclient(hunting): my global_id is 4135 2024-07-27T14:40:18.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.205+0000 7f2b65d9b640 1 -- 172.21.15.112:0/2563596753 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b50003520 con 0x7f2b680b2df0 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 1 -- 172.21.15.112:0/2563596753 >> v1:172.21.15.112:6789/0 conn(0x7f2b680b2df0 legacy=0x7f2b68140470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 1 -- 172.21.15.112:0/2563596753 >> v1:172.21.15.87:6789/0 conn(0x7f2b68072ff0 legacy=0x7f2b680aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 10 monclient: _finish_hunting 0 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 1 monclient: found mon.noname-a 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:40:18.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.206+0000 7f2b6659c640 1 -- 172.21.15.112:0/2563596753 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b680b4fd0 con 0x7f2b680b4650 2024-07-27T14:40:18.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.209+0000 7f2b6659c640 1 --2- 172.21.15.112:0/2563596753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b680b4650 0x7f2b6813ff30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09fb6b7a8ad697e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:18.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 1 -- 172.21.15.112:0/2563596753 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c02f810 con 0x7f2b680b4650 2024-07-27T14:40:18.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:40:18.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: dump: 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: _finish_auth 0 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: _check_auth_tickets 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 1 -- 172.21.15.112:0/2563596753 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5c02f9b0 con 0x7f2b680b4650 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 5 monclient: authenticate success, global_id 4134 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 4 set_mon_vals no callback set 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 1 -- 172.21.15.112:0/2563596753 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c02fd80 con 0x7f2b680b4650 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b65d9b640 10 monclient: dump: 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.216 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.217 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.217 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.217 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.217 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.217 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 10 monclient: shutdown 2024-07-27T14:40:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2563596753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b680b4650 msgr2=0x7f2b6813ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2563596753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b680b4650 0x7f2b6813ff30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b9f0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2563596753 shutdown_connections 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2563596753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b680b4650 0x7f2b6813ff30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2563596753 >> 172.21.15.112:0/2563596753 conn(0x7f2b680b8d70 msgr2=0x7f2b680b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2563596753 shutdown_connections 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.214+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2563596753 wait complete. 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.215+0000 7f2b6d0c5640 10 monclient: build_initial_monmap 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.215+0000 7f2b6d0c5640 10 monclient: monmap: 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.215+0000 7f2b6d0c5640 1 librados: starting msgr at 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 librados: starting objecter 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 Processor -- start 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 -- start start 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 librados: setting wanted keys 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 librados: calling monclient init 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient: init 2024-07-27T14:40:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b68072ff0 0x7f2b680acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2b68072ff0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b680b2df0 0x7f2b680aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2b680b2df0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2b680b4650 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6814e640 con 0x7f2b68072ff0 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2b680aacf0 con 0x7f2b680b4650 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6814e7c0 con 0x7f2b680b2df0 2024-07-27T14:40:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.216+0000 7f2b6d0c5640 10 monclient(hunting): authenticate will time out at 1159.245920s 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b66d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b68072ff0 0x7f2b680acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6659c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b680b2df0 0x7f2b680aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39194/0 (socket says 172.21.15.112:39194) 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 1 -- 172.21.15.112:0/2040606620 learned_addr learned my addr 172.21.15.112:0/2040606620 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 10 monclient(hunting): get_auth_request con 0x7f2b680b4650 auth_method 0 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6759e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b66d9d640 10 monclient(hunting): get_auth_request con 0x7f2b68072ff0 auth_method 0 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b66d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b66d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.217+0000 7f2b6659c640 10 monclient(hunting): get_auth_request con 0x7f2b680b2df0 auth_method 0 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b66d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b66d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b66d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.218+0000 7f2b6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 -- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b680b2df0 msgr2=0x7f2b680aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b680b2df0 0x7f2b680aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 -- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b68072ff0 msgr2=0x7f2b680acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b68072ff0 0x7f2b680acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 10 monclient: _finish_hunting 0 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 monclient: found mon.noname-b 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.221+0000 7f2b6759e640 1 -- 172.21.15.112:0/2040606620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b680af700 con 0x7f2b680b4650 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.223+0000 7f2b6759e640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000cae0 tx=0x7f2b6000a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.223+0000 7f2b4f7fe640 10 client.?.objecter ms_handle_connect 0x7f2b680b4650 2024-07-27T14:40:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.223+0000 7f2b4f7fe640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60017070 con 0x7f2b680b4650 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: dump: 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.229 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: _finish_auth 0 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: _check_auth_tickets 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6000ac40 con 0x7f2b680b4650 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 5 monclient: authenticate success, global_id 4138 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 10 monclient: _renew_subs 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b680ae580 con 0x7f2b680b4650 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60012640 con 0x7f2b680b4650 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b4f7fe640 10 monclient: dump: 2024-07-27T14:40:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 10 monclient: _renew_subs 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b680ac850 con 0x7f2b680b4650 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6559a640 4 set_mon_vals no callback set 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.226+0000 7f2b6d0c5640 1 librados: init done 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.227+0000 7f2b6d0c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.227+0000 7f2b6d0c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.227+0000 7f2b6d0c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.227+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b680aa460 con 0x7f2b680b4650 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2b6001a070 con 0x7f2b680b4650 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f2b60007580 con 0x7f2b680b4650 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 10 client.4138.objecter ms_dispatch 0x7f2b68000b90 osd_map(6..6 src has 1..6) 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 20 client.4138.objecter dump_active .. 0 homeless 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f2b60021020 con 0x7f2b680b4650 2024-07-27T14:40:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.327+0000 7f2b4f7fe640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:18.334 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.334+0000 7f7e10a15040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-07-27T14:40:18.394 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.394+0000 7f7e10a15040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-07-27T14:40:18.395 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.394+0000 7fc193f4f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-07-27T14:40:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.416+0000 7f2b6d0c5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a73fab50-80f0-4936-a1f4-91b6fd224114", "id": 4}] 2024-07-27T14:40:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.416+0000 7f2b6d0c5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a73fab50-80f0-4936-a1f4-91b6fd224114", "id": 4}] 2024-07-27T14:40:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.416+0000 7f2b6d0c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.416+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a73fab50-80f0-4936-a1f4-91b6fd224114", "id": 4} v 0) -- 0x7f2b680b17e0 con 0x7f2b680b4650 2024-07-27T14:40:18.448 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.445+0000 7fc193f4f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-07-27T14:40:18.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.485+0000 7f2b4f7fe640 1 -- 172.21.15.112:0/2040606620 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a73fab50-80f0-4936-a1f4-91b6fd224114", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f2b600078c0 con 0x7f2b680b4650 2024-07-27T14:40:18.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.485+0000 7f2b4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a73fab50-80f0-4936-a1f4-91b6fd224114", "id": 4}] 2024-07-27T14:40:18.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.485+0000 7f2b4f7fe640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:18.486 INFO:teuthology.orchestra.run.smithi112.stdout:4 2024-07-27T14:40:18.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 10 librados: watch_flush enter 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 10 librados: watch_flush exit 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 20 client.4138.objecter shutdown clearing up homeless session... 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 10 client.4138.objecter successfully canceled tick 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 10 monclient: shutdown 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 msgr2=0x7f2b680af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000cae0 tx=0x7f2b6000a060 comp rx=0 tx=0).stop 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 shutdown_connections 2024-07-27T14:40:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2b680b4650 0x7f2b680af1c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b680b2df0 0x7f2b680aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 --2- 172.21.15.112:0/2040606620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b68072ff0 0x7f2b680acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 >> 172.21.15.112:0/2040606620 conn(0x7f2b680b8d70 msgr2=0x7f2b6813e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 shutdown_connections 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 -- 172.21.15.112:0/2040606620 wait complete. 2024-07-27T14:40:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.486+0000 7f2b6d0c5640 1 librados: shutdown 2024-07-27T14:40:18.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560 5 2024-07-27T14:40:18.561 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.561+0000 7f7e10a15040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-07-27T14:40:18.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.595+0000 7f87285ec640 10 monclient: get_monmap_and_config 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.595+0000 7f87285ec640 10 monclient: build_initial_monmap 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.595+0000 7f87285ec640 10 monclient: monmap: 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:18.596537+0000 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:18.596537+0000 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:18.596 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:18.597 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 1 Processor -- start 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 1 -- start start 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 10 monclient: init 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87200b03b0 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 0x7f87200b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.596+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-b con 0x7f87200b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f87200b2310 0x7f872014d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-a con 0x7f87200b2310 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87200b4730 con 0x7f87200b03b0 2024-07-27T14:40:18.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f87200b0790 con 0x7f87200b2310 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f87200b2780 con 0x7f87200b58a0 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f87285ec640 10 monclient(hunting): authenticate will time out at 1159.626070s 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f8726b89640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f87200b2310 0x7f872014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f8726388640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f87200b03b0 0x7f8720140130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46282/0 (socket says 172.21.15.112:46282) 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f8726388640 1 -- 172.21.15.112:0/4256459556 learned_addr learned my addr 172.21.15.112:0/4256459556 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.597+0000 7f8725b87640 1 --2- 172.21.15.112:0/4256459556 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 0x7f87200b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): get_auth_request con 0x7f87200b58a0 auth_method 0 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): get_auth_request con 0x7f87200b2310 auth_method 0 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.598+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 10 monclient(hunting): handle_auth_done global_id 4130 payload 274 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 1 -- 172.21.15.112:0/4256459556 >> v2:172.21.15.87:3300/0 conn(0x7f87200b2310 msgr2=0x7f872014d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 1 --2- 172.21.15.112:0/4256459556 >> v2:172.21.15.87:3300/0 conn(0x7f87200b2310 0x7f872014d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 1 -- 172.21.15.112:0/4256459556 >> v1:172.21.15.112:6789/0 conn(0x7f87200b03b0 legacy=0x7f8720140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 10 monclient: _finish_hunting 0 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 1 monclient: found mon.noname-b 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:18.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:40:18.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.600+0000 7f8725b87640 1 -- 172.21.15.112:0/4256459556 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200b5380 con 0x7f87200b58a0 2024-07-27T14:40:18.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.602+0000 7f8725b87640 1 --2- 172.21.15.112:0/4256459556 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 0x7f87200b1dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f87100097a0 tx=0x7f871002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12564613f2ccd23e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:18.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 1 -- 172.21.15.112:0/4256459556 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87100315a0 con 0x7f87200b58a0 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: dump: 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.606 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: _finish_auth 0 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: _check_auth_tickets 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 1 -- 172.21.15.112:0/4256459556 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8710037070 con 0x7f87200b58a0 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 1 -- 172.21.15.112:0/4256459556 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710031960 con 0x7f87200b58a0 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f87285ec640 5 monclient: authenticate success, global_id 4130 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f87285ec640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f87285ec640 4 set_mon_vals no callback set 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.605+0000 7f8725386640 10 monclient: dump: 2024-07-27T14:40:18.607 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.608 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.609 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.607+0000 7fc193f4f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 10 monclient: shutdown 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 -- 172.21.15.112:0/4256459556 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 msgr2=0x7f87200b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 --2- 172.21.15.112:0/4256459556 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 0x7f87200b1dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f87100097a0 tx=0x7f871002b840 comp rx=0 tx=0).stop 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 -- 172.21.15.112:0/4256459556 shutdown_connections 2024-07-27T14:40:18.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 --2- 172.21.15.112:0/4256459556 >> v2:172.21.15.87:3300/0 conn(0x7f87200b2310 0x7f872014d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 --2- 172.21.15.112:0/4256459556 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b58a0 0x7f87200b1dd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 -- 172.21.15.112:0/4256459556 >> 172.21.15.112:0/4256459556 conn(0x7f8720073190 msgr2=0x7f8720067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 -- 172.21.15.112:0/4256459556 shutdown_connections 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.606+0000 7f87285ec640 1 -- 172.21.15.112:0/4256459556 wait complete. 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 10 monclient: build_initial_monmap 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 10 monclient: monmap: 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:18.613 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 librados: starting msgr at 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 librados: starting objecter 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 Processor -- start 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 -- start start 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 librados: setting wanted keys 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 1 librados: calling monclient init 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.607+0000 7f87285ec640 10 monclient: init 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87200b03b0 0x7f87200aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-a con 0x7f87200b03b0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b2310 0x7f8720146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-c con 0x7f87200b2310 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): picked mon.noname-b con 0x7f87200b58a0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f87200b3110 con 0x7f87200b03b0 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f87200b2f90 con 0x7f87200b58a0 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f87200b4f30 con 0x7f87200b2310 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f87285ec640 10 monclient(hunting): authenticate will time out at 1159.637240s 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f8726b89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f8725b87640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b2310 0x7f8720146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.608+0000 7f8726388640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87200b03b0 0x7f87200aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39196/0 (socket says 172.21.15.112:39196) 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 1 -- 172.21.15.112:0/4101370640 learned_addr learned my addr 172.21.15.112:0/4101370640 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): get_auth_request con 0x7f87200b58a0 auth_method 0 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): get_auth_request con 0x7f87200b2310 auth_method 0 2024-07-27T14:40:18.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): get_auth_request con 0x7f87200b03b0 auth_method 0 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8725b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.609+0000 7f8726388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 -- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b2310 msgr2=0x7f8720146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b2310 0x7f8720146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 -- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87200b03b0 msgr2=0x7f87200aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87200b03b0 0x7f87200aa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 10 monclient: _finish_hunting 0 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 monclient: found mon.noname-b 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.612+0000 7f8726b89640 1 -- 172.21.15.112:0/4101370640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200ac080 con 0x7f87200b58a0 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.614+0000 7f8726b89640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8718009dd0 tx=0x7f871800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.614+0000 7f8716ffd640 10 client.?.objecter ms_handle_connect 0x7f87200b58a0 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.614+0000 7f8716ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.617+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718018020 con 0x7f87200b58a0 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.617+0000 7f8716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.617+0000 7f8716ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.617+0000 7f8716ffd640 10 monclient: dump: 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: _finish_auth 0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8718014480 con 0x7f87200b58a0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 5 monclient: authenticate success, global_id 4144 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 10 monclient: _renew_subs 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8720146eb0 con 0x7f87200b58a0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718014870 con 0x7f87200b58a0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8716ffd640 10 monclient: dump: 2024-07-27T14:40:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f8724b85640 4 set_mon_vals no callback set 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 10 monclient: _renew_subs 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87201dbe30 con 0x7f87200b58a0 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.618+0000 7f87285ec640 1 librados: init done 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.619+0000 7f87285ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.619+0000 7f87285ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.619+0000 7f87285ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.619+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87200ab040 con 0x7f87200b58a0 2024-07-27T14:40:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8718005d10 con 0x7f87200b58a0 2024-07-27T14:40:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f8718013960 con 0x7f87200b58a0 2024-07-27T14:40:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 10 client.4144.objecter ms_dispatch 0x7f8720000b90 osd_map(7..7 src has 1..7) 2024-07-27T14:40:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-07-27T14:40:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-07-27T14:40:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.621+0000 7f8716ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-07-27T14:40:18.704 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.704+0000 7f7e10a15040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-07-27T14:40:18.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.718+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f87200ab040 con 0x7f87200b58a0 2024-07-27T14:40:18.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.718+0000 7f8716ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:18.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.718+0000 7f8716ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:18.748 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.747+0000 7fc193f4f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-07-27T14:40:18.757 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.757+0000 7f7e10a15040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-07-27T14:40:18.802 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.802+0000 7fc193f4f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-07-27T14:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.816+0000 7f87285ec640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560", "id": 5}] 2024-07-27T14:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.816+0000 7f87285ec640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560", "id": 5}] 2024-07-27T14:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.816+0000 7f87285ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.816+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560", "id": 5} v 0) -- 0x7f87200b1dd0 con 0x7f87200b58a0 2024-07-27T14:40:18.831 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.830+0000 7f7e10a15040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-07-27T14:40:18.877 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.877+0000 7fc193f4f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-07-27T14:40:18.878 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f7e10a15040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-07-27T14:40:18.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f8716ffd640 1 -- 172.21.15.112:0/4101370640 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f8718022070 con 0x7f87200b58a0 2024-07-27T14:40:18.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f8716ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560", "id": 5}] 2024-07-27T14:40:18.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f8716ffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:18.879 INFO:teuthology.orchestra.run.smithi112.stdout:5 2024-07-27T14:40:18.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f87285ec640 10 librados: watch_flush enter 2024-07-27T14:40:18.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f87285ec640 10 librados: watch_flush exit 2024-07-27T14:40:18.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f87285ec640 20 client.4144.objecter shutdown clearing up homeless session... 2024-07-27T14:40:18.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.878+0000 7f87285ec640 10 client.4144.objecter successfully canceled tick 2024-07-27T14:40:18.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 10 monclient: shutdown 2024-07-27T14:40:18.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:18.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 msgr2=0x7f87201db710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8718009dd0 tx=0x7f871800c490 comp rx=0 tx=0).stop 2024-07-27T14:40:18.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 shutdown_connections 2024-07-27T14:40:18.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87200b58a0 0x7f87201db710 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f87200b2310 0x7f8720146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 --2- 172.21.15.112:0/4101370640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87200b03b0 0x7f87200aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:18.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 >> 172.21.15.112:0/4101370640 conn(0x7f8720073190 msgr2=0x7f87200b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:18.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 shutdown_connections 2024-07-27T14:40:18.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 -- 172.21.15.112:0/4101370640 wait complete. 2024-07-27T14:40:18.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.879+0000 7f87285ec640 1 librados: shutdown 2024-07-27T14:40:18.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new 4b2984bd-8611-4a93-863a-852d8366fc3c 6 2024-07-27T14:40:18.924 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:18.924+0000 7fc193f4f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-07-27T14:40:18.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.988+0000 7fdd53844640 10 monclient: get_monmap_and_config 2024-07-27T14:40:18.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.988+0000 7fdd53844640 10 monclient: build_initial_monmap 2024-07-27T14:40:18.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.988+0000 7fdd53844640 10 monclient: monmap: 2024-07-27T14:40:18.989 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:18.989 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:18.989600+0000 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:18.989600+0000 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 1 Processor -- start 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 1 -- start start 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient: init 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdd4c0b03b0 0x7fdd4c140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c0b03b0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd4c0b58a0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd4c0b24e0 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:18.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.989+0000 7fdd53844640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd4c0b4730 con 0x7fdd4c0b58a0 2024-07-27T14:40:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd53844640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd4c0b5380 con 0x7fdd4c0b24e0 2024-07-27T14:40:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd53844640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd53844640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fdd4c0b0790 con 0x7fdd4c0b03b0 2024-07-27T14:40:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd53844640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd53844640 10 monclient(hunting): authenticate will time out at 1160.019031s 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd515e0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdd4c0b03b0 0x7fdd4c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd50ddf640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fdd4c0b58a0 0x7fdd4c0b1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.112:35946/0 (socket says 172.21.15.112:35946) 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.990+0000 7fdd50ddf640 1 -- 172.21.15.112:0/2224324103 learned_addr learned my addr 172.21.15.112:0/2224324103 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): get_auth_request con 0x7fdd4c0b03b0 auth_method 0 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:18.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:18.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:18.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:18.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:18.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.991+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 1 -- 172.21.15.112:0/2224324103 >> v1:172.21.15.112:6789/0 conn(0x7fdd4c0b24e0 legacy=0x7fdd4c14d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 1 -- 172.21.15.112:0/2224324103 >> v1:172.21.15.87:6789/0 conn(0x7fdd4c0b58a0 legacy=0x7fdd4c0b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 10 monclient: _finish_hunting 0 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 1 monclient: found mon.noname-c 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:18.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd515e0640 1 -- 172.21.15.112:0/2224324103 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c0afb90 con 0x7fdd4c0b03b0 2024-07-27T14:40:18.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd43fff640 1 -- 172.21.15.112:0/2224324103 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783306677 0 0) 0x7fdd4c0b4730 con 0x7fdd4c0b58a0 2024-07-27T14:40:18.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.993+0000 7fdd43fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-07-27T14:40:18.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:18.995+0000 7fdd515e0640 1 --2- 172.21.15.112:0/2224324103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c140130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8236bf4e27ba87a8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 1 -- 172.21.15.112:0/2224324103 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480315a0 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: dump: 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.000 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: _finish_auth 0 2024-07-27T14:40:19.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: _check_auth_tickets 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 1 -- 172.21.15.112:0/2224324103 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd48037070 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 5 monclient: authenticate success, global_id 4147 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 4 set_mon_vals no callback set 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 1 -- 172.21.15.112:0/2224324103 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48031960 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd43fff640 10 monclient: dump: 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.003 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 10 monclient: shutdown 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 1 -- 172.21.15.112:0/2224324103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 msgr2=0x7fdd4c140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 1 --2- 172.21.15.112:0/2224324103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c140130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b840 comp rx=0 tx=0).stop 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 1 -- 172.21.15.112:0/2224324103 shutdown_connections 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 1 --2- 172.21.15.112:0/2224324103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c140130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.000+0000 7fdd53844640 1 -- 172.21.15.112:0/2224324103 >> 172.21.15.112:0/2224324103 conn(0x7fdd4c073190 msgr2=0x7fdd4c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 1 -- 172.21.15.112:0/2224324103 shutdown_connections 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 1 -- 172.21.15.112:0/2224324103 wait complete. 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 10 monclient: build_initial_monmap 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 10 monclient: monmap: 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.009 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 1 librados: starting msgr at 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 1 librados: starting objecter 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.001+0000 7fdd53844640 1 Processor -- start 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 -- start start 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 librados: setting wanted keys 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 librados: calling monclient init 2024-07-27T14:40:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient: init 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4c0b03b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd4c0b24e0 0x7fdd4c0adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-a con 0x7fdd4c0b24e0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdd4c0b58a0 0x7fdd4c13cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c0b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c0b50b0 con 0x7fdd4c0b24e0 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c0b4f30 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4c140670 con 0x7fdd4c0b58a0 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.002+0000 7fdd53844640 10 monclient(hunting): authenticate will time out at 1160.031599s 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd50ddf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd4c0b24e0 0x7fdd4c0adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd51de1640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdd4c0b58a0 0x7fdd4c13cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39220/0 (socket says 172.21.15.112:39220) 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 1 -- 172.21.15.112:0/575542624 learned_addr learned my addr 172.21.15.112:0/575542624 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): get_auth_request con 0x7fdd4c0b03b0 auth_method 0 2024-07-27T14:40:19.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd50ddf640 10 monclient(hunting): get_auth_request con 0x7fdd4c0b24e0 auth_method 0 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd50ddf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd50ddf640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd50ddf640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd51de1640 10 monclient(hunting): get_auth_request con 0x7fdd4c0b58a0 auth_method 0 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd51de1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd51de1640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd51de1640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.003+0000 7fdd515e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd50ddf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd50ddf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd50ddf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd51de1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd51de1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.004+0000 7fdd51de1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 -- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdd4c0b58a0 msgr2=0x7fdd4c13cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdd4c0b58a0 0x7fdd4c13cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 -- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd4c0b24e0 msgr2=0x7fdd4c0adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd4c0b24e0 0x7fdd4c0adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 10 monclient: _finish_hunting 0 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 monclient: found mon.noname-b 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.006+0000 7fdd515e0640 1 -- 172.21.15.112:0/575542624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c0ae970 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.009+0000 7fdd515e0640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd48034040 tx=0x7fdd480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.009+0000 7fdd41ffb640 10 client.?.objecter ms_handle_connect 0x7fdd4c0b03b0 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.009+0000 7fdd41ffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48032840 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:19.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: dump: 2024-07-27T14:40:19.018 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.018 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.018 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.018 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.019 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.019 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.019 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.019 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.019 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: _finish_auth 0 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd480329e0 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 5 monclient: authenticate success, global_id 4150 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 10 monclient: _renew_subs 2024-07-27T14:40:19.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4c0aec40 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48032dd0 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd41ffb640 10 monclient: dump: 2024-07-27T14:40:19.021 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.022 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.022 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.022 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.022 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd437fe640 4 set_mon_vals no callback set 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:19.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 10 monclient: _renew_subs 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4c13d0f0 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.013+0000 7fdd53844640 1 librados: init done 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.014+0000 7fdd53844640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.014+0000 7fdd53844640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.014+0000 7fdd53844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.014+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd4c13fc00 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdd4804c070 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fdd4802f060 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 10 client.4150.objecter ms_dispatch 0x7fdd4c000b90 osd_map(8..8 src has 1..8) 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 20 client.4150.objecter dump_active .. 0 homeless 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7fdd4c13fc00 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.116+0000 7fdd41ffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.188+0000 7fdd53844640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4b2984bd-8611-4a93-863a-852d8366fc3c", "id": 6}] 2024-07-27T14:40:19.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.188+0000 7fdd53844640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4b2984bd-8611-4a93-863a-852d8366fc3c", "id": 6}] 2024-07-27T14:40:19.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.188+0000 7fdd53844640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.188+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4b2984bd-8611-4a93-863a-852d8366fc3c", "id": 6} v 0) -- 0x7fdd4c140130 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.250+0000 7fdd41ffb640 1 -- 172.21.15.112:0/575542624 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b2984bd-8611-4a93-863a-852d8366fc3c", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fdd4c0aec40 con 0x7fdd4c0b03b0 2024-07-27T14:40:19.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.250+0000 7fdd41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4b2984bd-8611-4a93-863a-852d8366fc3c", "id": 6}] 2024-07-27T14:40:19.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.250+0000 7fdd41ffb640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:19.251 INFO:teuthology.orchestra.run.smithi112.stdout:6 2024-07-27T14:40:19.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 10 librados: watch_flush enter 2024-07-27T14:40:19.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 10 librados: watch_flush exit 2024-07-27T14:40:19.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 20 client.4150.objecter shutdown clearing up homeless session... 2024-07-27T14:40:19.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 10 client.4150.objecter successfully canceled tick 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 10 monclient: shutdown 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 msgr2=0x7fdd4c0ad780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd48034040 tx=0x7fdd480046c0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 shutdown_connections 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdd4c0b58a0 0x7fdd4c13cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd4c0b24e0 0x7fdd4c0adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 --2- 172.21.15.112:0/575542624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd4c0b03b0 0x7fdd4c0ad780 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 >> 172.21.15.112:0/575542624 conn(0x7fdd4c073190 msgr2=0x7fdd4c0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 shutdown_connections 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 -- 172.21.15.112:0/575542624 wait complete. 2024-07-27T14:40:19.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.251+0000 7fdd53844640 1 librados: shutdown 2024-07-27T14:40:19.262 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd new c6c9e332-d670-435d-a37b-d6232ca7009c 7 2024-07-27T14:40:19.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.360+0000 7fe213bc3640 10 monclient: get_monmap_and_config 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.360+0000 7fe213bc3640 10 monclient: build_initial_monmap 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.360+0000 7fe213bc3640 10 monclient: monmap: 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:40:19.361380+0000 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:40:19.361380+0000 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:19.361 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 Processor -- start 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 -- start start 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient: init 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b03b0 0x7fe20c140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0b03b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fe20c0b58a0 0x7fe20c0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-c con 0x7fe20c0b58a0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe20c140670 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe20c0b4730 con 0x7fe20c140670 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fe20c0b0790 con 0x7fe20c0b58a0 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fe20c0b5380 con 0x7fe20c0b03b0 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.361+0000 7fe213bc3640 10 monclient(hunting): authenticate will time out at 1160.390913s 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21115e640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fe20c0b58a0 0x7fe20c0b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21195f640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b03b0 0x7fe20c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe212160640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fe20c140670 0x7fe20c0b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.112:35950/0 (socket says 172.21.15.112:35950) 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe212160640 1 -- 172.21.15.112:0/522077179 learned_addr learned my addr 172.21.15.112:0/522077179 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21115e640 10 monclient(hunting): get_auth_request con 0x7fe20c0b58a0 auth_method 0 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21115e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21115e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.362+0000 7fe21115e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): get_auth_request con 0x7fe20c0b03b0 auth_method 0 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.363+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 1 -- 172.21.15.112:0/522077179 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b03b0 msgr2=0x7fe20c140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:19.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 1 --2- 172.21.15.112:0/522077179 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b03b0 0x7fe20c140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 1 -- 172.21.15.112:0/522077179 >> v1:172.21.15.87:6789/0 conn(0x7fe20c140670 legacy=0x7fe20c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 10 monclient: _finish_hunting 0 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 1 monclient: found mon.noname-c 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.365+0000 7fe21115e640 1 -- 172.21.15.112:0/522077179 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c0afb90 con 0x7fe20c0b58a0 2024-07-27T14:40:19.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.367+0000 7fe21115e640 1 --2- 172.21.15.112:0/522077179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b58a0 0x7fe20c0b1dd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0097a0 tx=0x7fe1fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99cd2c90f31f7e38 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 1 -- 172.21.15.112:0/522077179 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0315a0 con 0x7fe20c0b58a0 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: dump: 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: _finish_auth 0 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: _check_auth_tickets 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 1 -- 172.21.15.112:0/522077179 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1fc037070 con 0x7fe20c0b58a0 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe213bc3640 5 monclient: authenticate success, global_id 4153 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe213bc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe213bc3640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe213bc3640 4 set_mon_vals no callback set 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 1 -- 172.21.15.112:0/522077179 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc031960 con 0x7fe20c0b58a0 2024-07-27T14:40:19.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.370+0000 7fe21095d640 10 monclient: dump: 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 10 monclient: shutdown 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 -- 172.21.15.112:0/522077179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b58a0 msgr2=0x7fe20c0b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 --2- 172.21.15.112:0/522077179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b58a0 0x7fe20c0b1dd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0097a0 tx=0x7fe1fc02b840 comp rx=0 tx=0).stop 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 -- 172.21.15.112:0/522077179 shutdown_connections 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 --2- 172.21.15.112:0/522077179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b58a0 0x7fe20c0b1dd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 --2- 172.21.15.112:0/522077179 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b03b0 0x7fe20c140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 -- 172.21.15.112:0/522077179 >> 172.21.15.112:0/522077179 conn(0x7fe20c073190 msgr2=0x7fe20c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 -- 172.21.15.112:0/522077179 shutdown_connections 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.371+0000 7fe213bc3640 1 -- 172.21.15.112:0/522077179 wait complete. 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.372+0000 7fe213bc3640 10 monclient: build_initial_monmap 2024-07-27T14:40:19.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.372+0000 7fe213bc3640 10 monclient: monmap: 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:19.378 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.372+0000 7fe213bc3640 1 librados: starting msgr at 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 librados: starting objecter 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 Processor -- start 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 -- start start 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 librados: setting wanted keys 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 librados: calling monclient init 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient: init 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0b03b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b58a0 0x7fe20c0aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-c con 0x7fe20c0b58a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe20c140670 0x7fe20c1dbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): picked mon.noname-a con 0x7fe20c140670 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c0b4f30 con 0x7fe20c140670 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c0b50b0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fe20c0b2310 con 0x7fe20c0b58a0 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.373+0000 7fe213bc3640 10 monclient(hunting): authenticate will time out at 1160.402699s 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe212160640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe20c140670 0x7fe20c1dbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21115e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b58a0 0x7fe20c0aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39244/0 (socket says 172.21.15.112:39244) 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 1 -- 172.21.15.112:0/581844430 learned_addr learned my addr 172.21.15.112:0/581844430 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 10 monclient(hunting): get_auth_request con 0x7fe20c0b03b0 auth_method 0 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21195f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe212160640 10 monclient(hunting): get_auth_request con 0x7fe20c140670 auth_method 0 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe212160640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe212160640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe212160640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21115e640 10 monclient(hunting): get_auth_request con 0x7fe20c0b58a0 auth_method 0 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21115e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21115e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.374+0000 7fe21115e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:19.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21195f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe212160640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe212160640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe212160640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.375+0000 7fe21115e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:19.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2024-07-27T14:40:19.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 -- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b58a0 msgr2=0x7fe20c0aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b58a0 0x7fe20c0aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 -- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe20c140670 msgr2=0x7fe20c1dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe20c140670 0x7fe20c1dbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 10 monclient: _finish_hunting 0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 monclient: found mon.noname-b 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.377+0000 7fe21195f640 1 -- 172.21.15.112:0/581844430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c0abf20 con 0x7fe20c0b03b0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.379+0000 7fe21195f640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e9e0 tx=0x7fe20800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.379+0000 7fe2027fc640 10 client.?.objecter ms_handle_connect 0x7fe20c0b03b0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.379+0000 7fe2027fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208017070 con 0x7fe20c0b03b0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: dump: 2024-07-27T14:40:19.385 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: _finish_auth 0 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: _check_auth_tickets 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe20800c9d0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 5 monclient: authenticate success, global_id 4156 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 10 monclient: _renew_subs 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20c140ef0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20800cdc0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe2027fc640 10 monclient: dump: 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe203fff640 4 set_mon_vals no callback set 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 10 monclient: _renew_subs 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20c1dc100 con 0x7fe20c0b03b0 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.383+0000 7fe213bc3640 1 librados: init done 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.384+0000 7fe213bc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.384+0000 7fe213bc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.384+0000 7fe213bc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.384+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe20c140130 con 0x7fe20c0b03b0 2024-07-27T14:40:19.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.386+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe20801a070 con 0x7fe20c0b03b0 2024-07-27T14:40:19.389 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:19.389+0000 7f7e10a15040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-07-27T14:40:19.412 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:19.412+0000 7fc193f4f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-07-27T14:40:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe208005ff0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 10 client.4156.objecter ms_dispatch 0x7fe20c000b90 osd_map(9..9 src has 1..9) 2024-07-27T14:40:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-07-27T14:40:19.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-07-27T14:40:19.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 20 client.4156.objecter dump_active .. 0 homeless 2024-07-27T14:40:19.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7fe2080065d0 con 0x7fe20c0b03b0 2024-07-27T14:40:19.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:19.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.484+0000 7fe2027fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:19.510 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:19.509+0000 7f7e10a15040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-07-27T14:40:19.529 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-07-27T14:40:19.528+0000 7fc193f4f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-07-27T14:40:19.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.556+0000 7fe213bc3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c6c9e332-d670-435d-a37b-d6232ca7009c", "id": 7}] 2024-07-27T14:40:19.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.556+0000 7fe213bc3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c6c9e332-d670-435d-a37b-d6232ca7009c", "id": 7}] 2024-07-27T14:40:19.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.556+0000 7fe213bc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:19.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.556+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c6c9e332-d670-435d-a37b-d6232ca7009c", "id": 7} v 0) -- 0x7fe20c0b7790 con 0x7fe20c0b03b0 2024-07-27T14:40:19.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:19.807+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7fe208005990 con 0x7fe20c0b03b0 2024-07-27T14:40:20.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.009+0000 7fe2027fc640 1 -- 172.21.15.112:0/581844430 <== mon.1 v2:172.21.15.112:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6c9e332-d670-435d-a37b-d6232ca7009c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe2080203a0 con 0x7fe20c0b03b0 2024-07-27T14:40:20.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.009+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c6c9e332-d670-435d-a37b-d6232ca7009c", "id": 7}] 2024-07-27T14:40:20.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.009+0000 7fe2027fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:20.010 INFO:teuthology.orchestra.run.smithi112.stdout:7 2024-07-27T14:40:20.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 10 librados: watch_flush enter 2024-07-27T14:40:20.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 10 librados: watch_flush exit 2024-07-27T14:40:20.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 20 client.4156.objecter shutdown clearing up homeless session... 2024-07-27T14:40:20.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 10 client.4156.objecter successfully canceled tick 2024-07-27T14:40:20.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 10 monclient: shutdown 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 msgr2=0x7fe20c0aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e9e0 tx=0x7fe20800eea0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 shutdown_connections 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe20c140670 0x7fe20c1dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe20c0b58a0 0x7fe20c0aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 --2- 172.21.15.112:0/581844430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe20c0b03b0 0x7fe20c0aa6d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 >> 172.21.15.112:0/581844430 conn(0x7fe20c073190 msgr2=0x7fe20c0b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 shutdown_connections 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 -- 172.21.15.112:0/581844430 wait complete. 2024-07-27T14:40:20.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-27T14:40:20.010+0000 7fe213bc3640 1 librados: shutdown 2024-07-27T14:40:20.021 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.022 INFO:tasks.ceph.osd.0:Restarting daemon 2024-07-27T14:40:20.022 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-07-27T14:40:20.024 INFO:tasks.ceph.osd.0:Started 2024-07-27T14:40:20.024 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.024 INFO:tasks.ceph.osd.1:Restarting daemon 2024-07-27T14:40:20.024 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-07-27T14:40:20.026 INFO:tasks.ceph.osd.1:Started 2024-07-27T14:40:20.027 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.027 INFO:tasks.ceph.osd.2:Restarting daemon 2024-07-27T14:40:20.027 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-07-27T14:40:20.029 INFO:tasks.ceph.osd.2:Started 2024-07-27T14:40:20.029 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.029 INFO:tasks.ceph.osd.3:Restarting daemon 2024-07-27T14:40:20.029 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-07-27T14:40:20.031 INFO:tasks.ceph.osd.3:Started 2024-07-27T14:40:20.031 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.032 INFO:tasks.ceph.osd.4:Restarting daemon 2024-07-27T14:40:20.032 DEBUG:teuthology.orchestra.run.smithi112:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-07-27T14:40:20.034 INFO:tasks.ceph.osd.4:Started 2024-07-27T14:40:20.034 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.034 INFO:tasks.ceph.osd.5:Restarting daemon 2024-07-27T14:40:20.034 DEBUG:teuthology.orchestra.run.smithi112:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-07-27T14:40:20.036 INFO:tasks.ceph.osd.5:Started 2024-07-27T14:40:20.036 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.036 INFO:tasks.ceph.osd.6:Restarting daemon 2024-07-27T14:40:20.037 DEBUG:teuthology.orchestra.run.smithi112:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-07-27T14:40:20.039 INFO:tasks.ceph.osd.6:Started 2024-07-27T14:40:20.039 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-27T14:40:20.039 INFO:tasks.ceph.osd.7:Restarting daemon 2024-07-27T14:40:20.039 DEBUG:teuthology.orchestra.run.smithi112:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-07-27T14:40:20.042 INFO:tasks.ceph.osd.7:Started 2024-07-27T14:40:20.043 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:20.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.197+0000 7f1a63ebb640 10 monclient: get_monmap_and_config 2024-07-27T14:40:20.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.197+0000 7f1a63ebb640 10 monclient: build_initial_monmap 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.197+0000 7f1a63ebb640 10 monclient: monmap: 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:40:20.198403+0000 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:40:20.198403+0000 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:20.198 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:20.199 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.198+0000 7f1a63ebb640 1 Processor -- start 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.198+0000 7f1a63ebb640 1 -- start start 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.198+0000 7f1a63ebb640 10 monclient: init 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-07-27T14:40:20.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a5c0b3630 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f1a5c0aab60 0x7f1a5c13ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c0aab60 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1a5c140450 0x7f1a5c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5c140450 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5c0aa8a0 con 0x7f1a5c0b3630 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f1a5c0b3a10 con 0x7f1a5c0aab60 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f1a5c142da0 con 0x7f1a5c140450 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.199+0000 7f1a63ebb640 10 monclient(hunting): authenticate will time out at 1160.225414s 2024-07-27T14:40:20.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1a5c140450 0x7f1a5c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:20.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1a5c140450 0x7f1a5c14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:55230/0 (socket says 172.21.15.87:55230) 2024-07-27T14:40:20.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 1 -- 172.21.15.87:0/3823740875 learned_addr learned my addr 172.21.15.87:0/3823740875 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:20.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 10 monclient(hunting): get_auth_request con 0x7f1a5c140450 auth_method 0 2024-07-27T14:40:20.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.200+0000 7f1a62458640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.201+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.201+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.201+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.201+0000 7f1a61456640 1 --2- 172.21.15.87:0/3823740875 >> v2:172.21.15.87:3300/0 conn(0x7f1a5c0aab60 0x7f1a5c13ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.202+0000 7f1a61456640 10 monclient(hunting): get_auth_request con 0x7f1a5c0aab60 auth_method 0 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.202+0000 7f1a61456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.202+0000 7f1a61456640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:20.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.202+0000 7f1a61456640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:20.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 10 monclient(hunting): handle_auth_done global_id 4168 payload 274 2024-07-27T14:40:20.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 1 -- 172.21.15.87:0/3823740875 >> v2:172.21.15.87:3300/0 conn(0x7f1a5c0aab60 msgr2=0x7f1a5c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 1 --2- 172.21.15.87:0/3823740875 >> v2:172.21.15.87:3300/0 conn(0x7f1a5c0aab60 0x7f1a5c13ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 1 -- 172.21.15.87:0/3823740875 >> v1:172.21.15.87:6789/0 conn(0x7f1a5c0b3630 legacy=0x7f1a5c143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 10 monclient: _finish_hunting 0 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 1 monclient: found mon.noname-c 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:20.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.204+0000 7f1a62458640 1 -- 172.21.15.87:0/3823740875 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5c143940 con 0x7f1a5c140450 2024-07-27T14:40:20.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.207+0000 7f1a62458640 1 --2- 172.21.15.87:0/3823740875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c140450 0x7f1a5c14d410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0b62f0 tx=0x7f1a5402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=347bbd1bfd8ab73a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.209+0000 7f1a60c55640 1 -- 172.21.15.87:0/3823740875 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5402c9f0 con 0x7f1a5c140450 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 10 monclient: dump: 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 10 monclient: _finish_auth 0 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 10 monclient: _check_auth_tickets 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a60c55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a63ebb640 5 monclient: authenticate success, global_id 4168 2024-07-27T14:40:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.210+0000 7f1a63ebb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 1 -- 172.21.15.87:0/3823740875 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5402cb90 con 0x7f1a5c140450 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 1 -- 172.21.15.87:0/3823740875 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a540343f0 con 0x7f1a5c140450 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a60c55640 10 monclient: dump: 2024-07-27T14:40:20.218 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a63ebb640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a63ebb640 4 set_mon_vals no callback set 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a63ebb640 10 monclient: shutdown 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a63ebb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:20.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.212+0000 7f1a63ebb640 1 -- 172.21.15.87:0/3823740875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c140450 msgr2=0x7f1a5c14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/3823740875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c140450 0x7f1a5c14d410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0b62f0 tx=0x7f1a5402be50 comp rx=0 tx=0).stop 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 -- 172.21.15.87:0/3823740875 shutdown_connections 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/3823740875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c140450 0x7f1a5c14d410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/3823740875 >> v2:172.21.15.87:3300/0 conn(0x7f1a5c0aab60 0x7f1a5c13ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 -- 172.21.15.87:0/3823740875 >> 172.21.15.87:0/3823740875 conn(0x7f1a5c0730a0 msgr2=0x7f1a5c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 -- 172.21.15.87:0/3823740875 shutdown_connections 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 1 -- 172.21.15.87:0/3823740875 wait complete. 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.213+0000 7f1a63ebb640 10 monclient: build_initial_monmap 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 10 monclient: monmap: 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:40:20.220 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 librados: starting msgr at 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 librados: starting objecter 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 Processor -- start 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 -- start start 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 librados: setting wanted keys 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 1 librados: calling monclient init 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.214+0000 7f1a63ebb640 10 monclient: init 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:20.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5c0aab60 0x7f1a5c0ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c0aab60 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c0b3630 0x7f1a5c1cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5c0b3630 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5c140450 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5c0b4840 con 0x7f1a5c0aab60 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5c0b46c0 con 0x7f1a5c0b3630 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1a5c0b49c0 con 0x7f1a5c140450 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:20.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a63ebb640 10 monclient(hunting): authenticate will time out at 1160.241243s 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.215+0000 7f1a62458640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a61456640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c0b3630 0x7f1a5c1cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48304/0 (socket says 172.21.15.87:48304) 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 1 -- 172.21.15.87:0/1669516942 learned_addr learned my addr 172.21.15.87:0/1669516942 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): get_auth_request con 0x7f1a5c140450 auth_method 0 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a61456640 10 monclient(hunting): get_auth_request con 0x7f1a5c0b3630 auth_method 0 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a61456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a61456640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a61456640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:20.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:20.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.216+0000 7f1a62458640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:20.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.217+0000 7f1a61456640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:20.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.217+0000 7f1a61456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:20.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.217+0000 7f1a61456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 -- 172.21.15.87:0/1669516942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c0b3630 msgr2=0x7f1a5c1cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c0b3630 0x7f1a5c1cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 -- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5c0aab60 msgr2=0x7f1a5c0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5c0aab60 0x7f1a5c0ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 10 monclient: _finish_hunting 0 2024-07-27T14:40:20.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 monclient: found mon.noname-c 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.220+0000 7f1a62458640 1 -- 172.21.15.87:0/1669516942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5c0adbb0 con 0x7f1a5c140450 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.223+0000 7f1a62458640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c460 tx=0x7f1a54004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.223+0000 7f1a527fc640 10 client.?.objecter ms_handle_connect 0x7f1a5c140450 2024-07-27T14:40:20.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.223+0000 7f1a527fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54043020 con 0x7f1a5c140450 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: dump: 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:20.245 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: _finish_auth 0 2024-07-27T14:40:20.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: _check_auth_tickets 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5403e450 con 0x7f1a5c140450 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a63ebb640 5 monclient: authenticate success, global_id 4160 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a63ebb640 10 monclient: _renew_subs 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a63ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a5c0ade90 con 0x7f1a5c140450 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54009510 con 0x7f1a5c140450 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a527fc640 10 monclient: dump: 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:40:20.247 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.243+0000 7f1a53fff640 4 set_mon_vals no callback set 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.244+0000 7f1a63ebb640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.244+0000 7f1a63ebb640 10 monclient: _renew_subs 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.244+0000 7f1a63ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:20.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.244+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a5c0ad4d0 con 0x7f1a5c140450 2024-07-27T14:40:20.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.244+0000 7f1a63ebb640 1 librados: init done 2024-07-27T14:40:20.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.245+0000 7f1a63ebb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:20.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.245+0000 7f1a63ebb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:20.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.245+0000 7f1a63ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:20.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.245+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a5c0acde0 con 0x7f1a5c140450 2024-07-27T14:40:20.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.429+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7f1a54005d10 con 0x7f1a5c140450 2024-07-27T14:40:20.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1a5403e820 con 0x7f1a5c140450 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 10 client.4160.objecter ms_dispatch 0x7f1a5c000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 3 client.4160.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 20 client.4160.objecter dump_active .. 0 homeless 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73327 (secure 0 0 0) 0x7f1a54052480 con 0x7f1a5c140450 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:20.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.437+0000 7f1a527fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:20.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.503+0000 7f1a63ebb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:20.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.503+0000 7f1a63ebb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:20.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.503+0000 7f1a63ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:20.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.503+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a5c13ff10 con 0x7f1a5c140450 2024-07-27T14:40:20.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.614+0000 7f1a527fc640 1 -- 172.21.15.87:0/1669516942 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1a5403eab0 con 0x7f1a5c140450 2024-07-27T14:40:20.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.614+0000 7f1a527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:20.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.614+0000 7f1a527fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:40:20.615 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 10 librados: watch_flush enter 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 10 librados: watch_flush exit 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 20 client.4160.objecter shutdown clearing up homeless session... 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 10 client.4160.objecter successfully canceled tick 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 10 monclient: shutdown 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 msgr2=0x7f1a5c0b18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c460 tx=0x7f1a54004a40 comp rx=0 tx=0).stop 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 shutdown_connections 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1a5c140450 0x7f1a5c0b18f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a5c0b3630 0x7f1a5c1cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 --2- 172.21.15.87:0/1669516942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5c0aab60 0x7f1a5c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:20.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 >> 172.21.15.87:0/1669516942 conn(0x7f1a5c0730a0 msgr2=0x7f1a5c143230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:20.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 shutdown_connections 2024-07-27T14:40:20.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 -- 172.21.15.87:0/1669516942 wait complete. 2024-07-27T14:40:20.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:40:20.615+0000 7f1a63ebb640 1 librados: shutdown 2024-07-27T14:40:20.630 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-07-27T14:40:20.631 INFO:tasks.ceph:Waiting for OSDs to come up 2024-07-27T14:40:21.437 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:21.582 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.581+0000 7fa57f984640 10 monclient: get_monmap_and_config 2024-07-27T14:40:21.582 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.581+0000 7fa57f984640 10 monclient: build_initial_monmap 2024-07-27T14:40:21.582 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.582+0000 7fa57f984640 10 monclient: monmap: 2024-07-27T14:40:21.582 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:40:21.583038+0000 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:40:21.583038+0000 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:21.583 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.582+0000 7fa57f984640 1 Processor -- start 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.583+0000 7fa57f984640 1 -- start start 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.583+0000 7fa57f984640 10 monclient: init 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-07-27T14:40:21.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa5780b92d0 0x7fa5780b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-c con 0x7fa5780b92d0 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5780b2d80 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa5780aaa10 0x7fa5781460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-a con 0x7fa5780aaa10 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5780aae80 con 0x7fa5780b2d80 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fa5780b96b0 con 0x7fa5780aaa10 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fa5780b5230 con 0x7fa5780b92d0 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:21.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.584+0000 7fa57f984640 10 monclient(hunting): authenticate will time out at 1161.610290s 2024-07-27T14:40:21.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.586+0000 7fa57d720640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa5780b92d0 0x7fa5780b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:21.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa5780b92d0 0x7fa5780b4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:55244/0 (socket says 172.21.15.87:55244) 2024-07-27T14:40:21.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 1 -- 172.21.15.87:0/2792691250 learned_addr learned my addr 172.21.15.87:0/2792691250 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:21.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 10 monclient(hunting): get_auth_request con 0x7fa5780b92d0 auth_method 0 2024-07-27T14:40:21.588 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:21.588 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:21.588 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.587+0000 7fa57d720640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:21.588 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.588+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:21.589 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.588+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:21.589 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.588+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:21.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 10 monclient(hunting): handle_auth_done global_id 4174 payload 274 2024-07-27T14:40:21.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 1 -- 172.21.15.87:0/2792691250 >> v2:172.21.15.87:3300/0 conn(0x7fa5780aaa10 msgr2=0x7fa5781460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:21.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 1 --2- 172.21.15.87:0/2792691250 >> v2:172.21.15.87:3300/0 conn(0x7fa5780aaa10 0x7fa5781460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:21.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 1 -- 172.21.15.87:0/2792691250 >> v1:172.21.15.87:6789/0 conn(0x7fa5780b2d80 legacy=0x7fa578140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:21.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 10 monclient: _finish_hunting 0 2024-07-27T14:40:21.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 1 monclient: found mon.noname-c 2024-07-27T14:40:21.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:21.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:21.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.603+0000 7fa57d720640 1 -- 172.21.15.87:0/2792691250 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5780b4640 con 0x7fa5780b92d0 2024-07-27T14:40:21.613 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.613+0000 7fa57d720640 1 --2- 172.21.15.87:0/2792691250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b92d0 0x7fa5780b4b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa574009870 tx=0x7fa57402b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=325a977ae9b47181 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:21.658 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:21.658+0000 7f7db09ae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-27T14:40:21.669 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 1 -- 172.21.15.87:0/2792691250 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574031760 con 0x7fa5780b92d0 2024-07-27T14:40:21.669 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:21.669 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:21.669 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: dump: 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: _finish_auth 0 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: _check_auth_tickets 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:21.670 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 1 -- 172.21.15.87:0/2792691250 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa574037070 con 0x7fa5780b92d0 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 1 -- 172.21.15.87:0/2792691250 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574031b60 con 0x7fa5780b92d0 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa56ffff640 10 monclient: dump: 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:21.671 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:21.672 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa57f984640 5 monclient: authenticate success, global_id 4174 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa57f984640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.669+0000 7fa57f984640 4 set_mon_vals no callback set 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 10 monclient: shutdown 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 -- 172.21.15.87:0/2792691250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b92d0 msgr2=0x7fa5780b4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 --2- 172.21.15.87:0/2792691250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b92d0 0x7fa5780b4b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa574009870 tx=0x7fa57402b800 comp rx=0 tx=0).stop 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 -- 172.21.15.87:0/2792691250 shutdown_connections 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 --2- 172.21.15.87:0/2792691250 >> v2:172.21.15.87:3300/0 conn(0x7fa5780aaa10 0x7fa5781460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 --2- 172.21.15.87:0/2792691250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b92d0 0x7fa5780b4b60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:21.675 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 -- 172.21.15.87:0/2792691250 >> 172.21.15.87:0/2792691250 conn(0x7fa578067d80 msgr2=0x7fa5780b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 -- 172.21.15.87:0/2792691250 shutdown_connections 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.670+0000 7fa57f984640 1 -- 172.21.15.87:0/2792691250 wait complete. 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.671+0000 7fa57f984640 10 monclient: build_initial_monmap 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.671+0000 7fa57f984640 10 monclient: monmap: 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:21.676 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.671+0000 7fa57f984640 1 librados: starting msgr at 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.671+0000 7fa57f984640 1 librados: starting objecter 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 1 Processor -- start 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 1 -- start start 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 1 librados: setting wanted keys 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 1 librados: calling monclient init 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 10 monclient: init 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.672+0000 7fa57f984640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5780aaa10 0x7fa57814b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-a con 0x7fa5780aaa10 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:21.677 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-b con 0x7fa5780b2d80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5780b92d0 0x7fa5781489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): picked mon.noname-c con 0x7fa5780b92d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa578146800 con 0x7fa5780aaa10 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa57814d3e0 con 0x7fa5780b2d80 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa57814be00 con 0x7fa5780b92d0 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57cf1f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.673+0000 7fa57f984640 10 monclient(hunting): authenticate will time out at 1161.699913s 2024-07-27T14:40:21.678 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:55246/0 (socket says 172.21.15.87:55246) 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 1 -- 172.21.15.87:0/1032261303 learned_addr learned my addr 172.21.15.87:0/1032261303 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 10 monclient(hunting): get_auth_request con 0x7fa5780b2d80 auth_method 0 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.674+0000 7fa57cf1f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:21.679 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.675+0000 7fa57cf1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.675+0000 7fa57cf1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.675+0000 7fa57cf1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.677+0000 7fa57d720640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5780aaa10 0x7fa57814b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.677+0000 7fa57d720640 10 monclient(hunting): get_auth_request con 0x7fa5780aaa10 auth_method 0 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.677+0000 7fa57d720640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.677+0000 7fa57d720640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.677+0000 7fa57d720640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.678+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.678+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:21.680 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.678+0000 7fa57d720640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 10 monclient(hunting): handle_auth_done global_id 4177 payload 824 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 -- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5780b92d0 msgr2=0x7fa5781489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5780b92d0 0x7fa5781489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 -- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5780aaa10 msgr2=0x7fa57814b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5780aaa10 0x7fa57814b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 10 monclient: _finish_hunting 0 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 monclient: found mon.noname-b 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:21.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.679+0000 7fa57cf1f640 1 -- 172.21.15.87:0/1032261303 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57814cc60 con 0x7fa5780b2d80 2024-07-27T14:40:21.683 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.683+0000 7fa57cf1f640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa56800c410 tx=0x7fa56800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:21.683 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.683+0000 7fa56dffb640 10 client.?.objecter ms_handle_connect 0x7fa5780b2d80 2024-07-27T14:40:21.683 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.683+0000 7fa56dffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568017070 con 0x7fa5780b2d80 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 10 monclient: dump: 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:21.687 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 10 monclient: _finish_auth 0 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 10 monclient: _check_auth_tickets 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa56dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 5 monclient: authenticate success, global_id 4177 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 10 monclient: _renew_subs 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57814cf40 con 0x7fa5780b2d80 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 10 monclient: _renew_subs 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57814b9b0 con 0x7fa5780b2d80 2024-07-27T14:40:21.688 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.686+0000 7fa57f984640 1 librados: init done 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa57f984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa57f984640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa57f984640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5781460b0 con 0x7fa5780b2d80 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa568007020 con 0x7fa5780b2d80 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568021060 con 0x7fa5780b2d80 2024-07-27T14:40:21.689 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56dffb640 10 monclient: dump: 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:21.690 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:21.688+0000 7fa56f7fe640 4 set_mon_vals no callback set 2024-07-27T14:40:21.739 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T14:40:21.739+0000 7f7db09ae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-27T14:40:22.126 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.125+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 3) ==== 45401+0+0 (secure 0 0 0) 0x7fa56801a070 con 0x7fa5780b2d80 2024-07-27T14:40:22.126 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.125+0000 7fa56dffb640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 0x7fa55c03be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:22.126 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.126+0000 7fa57d720640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 0x7fa55c03be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:22.126 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.126+0000 7fa57d720640 10 monclient: get_auth_request con 0x7fa55c039950 auth_method 0 2024-07-27T14:40:22.127 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.126+0000 7fa56dffb640 10 client.4177.objecter ms_handle_connect 0x7fa55c039950 2024-07-27T14:40:22.127 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.126+0000 7fa57d720640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 0x7fa55c03be10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:22.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.137+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa56801b570 con 0x7fa5780b2d80 2024-07-27T14:40:22.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.137+0000 7fa56dffb640 10 client.4177.objecter ms_dispatch 0x7fa578000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:22.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.137+0000 7fa56dffb640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:22.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.137+0000 7fa56dffb640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:22.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.137+0000 7fa56dffb640 20 client.4177.objecter dump_active .. 0 homeless 2024-07-27T14:40:22.164 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.164+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fa5680075b0 con 0x7fa5780b2d80 2024-07-27T14:40:22.164 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.164+0000 7fa56dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:22.164 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.164+0000 7fa56dffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:22.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.540+0000 7fa57f984640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:22.541 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.540+0000 7fa57f984640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:22.541 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.540+0000 7fa57f984640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:22.541 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.540+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5780ab080 con 0x7fa5780b2d80 2024-07-27T14:40:22.889 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:22.888+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fa568006cd0 con 0x7fa5780b2d80 2024-07-27T14:40:23.020 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.019+0000 7fa56dffb640 1 -- 172.21.15.87:0/1032261303 <== mon.1 v2:172.21.15.112:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa568046020 con 0x7fa5780b2d80 2024-07-27T14:40:23.020 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.019+0000 7fa56dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:23.020 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.019+0000 7fa56dffb640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:23.020 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:40:23.020 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:23.024 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.023+0000 7fa57f984640 10 librados: watch_flush enter 2024-07-27T14:40:23.024 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 10 librados: watch_flush exit 2024-07-27T14:40:23.024 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 20 client.4177.objecter shutdown clearing up homeless session... 2024-07-27T14:40:23.024 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 10 client.4177.objecter successfully canceled tick 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 msgr2=0x7fa55c03be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 0x7fa55c03be10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403d040 comp rx=0 tx=0).stop 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 10 monclient: shutdown 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 msgr2=0x7fa5781484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa56800c410 tx=0x7fa56800c8d0 comp rx=0 tx=0).stop 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 shutdown_connections 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa55c039950 0x7fa55c03be10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:23.025 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5780b92d0 0x7fa5781489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5780b2d80 0x7fa5781484b0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa56800c410 tx=0x7fa56800c8d0 comp rx=0 tx=0).stop 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 --2- 172.21.15.87:0/1032261303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5780aaa10 0x7fa57814b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 >> 172.21.15.87:0/1032261303 conn(0x7fa578067d80 msgr2=0x7fa578140030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 shutdown_connections 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.024+0000 7fa57f984640 1 -- 172.21.15.87:0/1032261303 wait complete. 2024-07-27T14:40:23.026 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:23.025+0000 7fa57f984640 1 librados: shutdown 2024-07-27T14:40:23.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-27T14:40:27.298 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:07.185 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.324 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:07.217 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.363 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:07.250 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.384 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:07.277 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.472 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:07.355 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.532 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:07.415 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.758 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:07.660 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.815 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:07.717 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.846 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:07.735 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.910 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:40:27.903+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:27.915 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:07.813 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.923 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:07.812 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.936 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:40:27.930+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:27.973 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:07.854 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:27.976 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:07.874 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.028 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:07.915 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.032 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:07.914 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.091 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:07.977 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.100 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:40:28.095+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.342 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:08.230 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.389 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:08.282 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.413 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:40:28.405+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:40:28.463+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:40:28.526+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.606 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:08.489 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.621 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:40:28.613+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:40:28.634+0000 5dce500 -1 Falling back to public interface 2024-07-27T14:40:28.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:40:28.646+0000 8c80640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6819/0: (98) Address already in use 2024-07-27T14:40:28.709 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:08.597 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.763 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:08.656 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.872 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:08.774 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.911 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:08.799 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:28.974 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:08.857 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.040 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:08.938 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.068 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:08.955 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.080 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:08.973 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.107 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:08.993 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.112 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:08.994 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.242 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:09.144 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.281 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.170 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.293 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:09.177 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.348 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:09.242 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.366 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:09.253 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.404 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.301 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.481 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.367 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.485 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.367 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.566 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:09.468 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.568 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:09.451 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.596 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.484 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.613 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:09.507 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.654 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:09.541 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.717 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.614 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.810 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.691 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.834 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:09.736 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.841 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:09.725 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.843 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.729 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.848 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:29.878 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.766 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.884 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:09.778 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.940 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.940+0000 7f75aa873640 10 monclient: get_monmap_and_config 2024-07-27T14:40:29.940 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.940+0000 7f75aa873640 10 monclient: build_initial_monmap 2024-07-27T14:40:29.940 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.940+0000 7f75aa873640 10 monclient: monmap: 2024-07-27T14:40:29.940 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:29.940 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:40:29.941096+0000 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:40:29.941096+0000 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:29.941 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.943 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.940+0000 7f75aa873640 1 Processor -- start 2024-07-27T14:40:29.943 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 -- start start 2024-07-27T14:40:29.943 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient: init 2024-07-27T14:40:29.943 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:29.943 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75a41463d0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f75a4140610 0x7f75a41467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-c con 0x7f75a4140610 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-b con 0x7f75a40aaa30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75a40aae70 con 0x7f75a41463d0 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f75a40abe30 con 0x7f75a4140610 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f75a4146e50 con 0x7f75a40aaa30 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.941+0000 7f75aa873640 10 monclient(hunting): authenticate will time out at 1169.967746s 2024-07-27T14:40:29.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f75a3fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f75a41463d0 0x7f75a413ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:33568/0 (socket says 172.21.15.87:33568) 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f75a3fff640 1 -- 172.21.15.87:0/3661571470 learned_addr learned my addr 172.21.15.87:0/3661571470 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f759bfff640 1 --2- 172.21.15.87:0/3661571470 >> v2:172.21.15.112:3300/0 conn(0x7f75a4140610 0x7f75a41467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f759bfff640 10 monclient(hunting): get_auth_request con 0x7f75a4140610 auth_method 0 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f759bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f759bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f759bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.942+0000 7f75a8e10640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): get_auth_request con 0x7f75a40aaa30 auth_method 0 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:29.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.943+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 10 monclient(hunting): handle_auth_done global_id 4204 payload 274 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 1 -- 172.21.15.87:0/3661571470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 msgr2=0x7f75a40b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 1 -- 172.21.15.87:0/3661571470 >> v1:172.21.15.87:6789/0 conn(0x7f75a41463d0 legacy=0x7f75a413ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 10 monclient: _finish_hunting 0 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 1 monclient: found mon.noname-c 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.944+0000 7f759bfff640 1 -- 172.21.15.87:0/3661571470 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a40b3810 con 0x7f75a4140610 2024-07-27T14:40:29.947 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.946+0000 7f759bfff640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a4140610 0x7f75a41467b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f758c0097a0 tx=0x7f758c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c9ae84969a616ec server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:29.949 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:09.836 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 1 -- 172.21.15.87:0/3661571470 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c0315a0 con 0x7f75a4140610 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: dump: 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:29.951 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: _finish_auth 0 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: _check_auth_tickets 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 1 -- 172.21.15.87:0/3661571470 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f758c037070 con 0x7f75a4140610 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 1 -- 172.21.15.87:0/3661571470 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c031960 con 0x7f75a4140610 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 5 monclient: authenticate success, global_id 4204 2024-07-27T14:40:29.952 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 4 set_mon_vals no callback set 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75a37fe640 10 monclient: dump: 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:29.953 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 10 monclient: shutdown 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 -- 172.21.15.87:0/3661571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a4140610 msgr2=0x7f75a41467b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a4140610 0x7f75a41467b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f758c0097a0 tx=0x7f758c02b840 comp rx=0 tx=0).stop 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 -- 172.21.15.87:0/3661571470 shutdown_connections 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 --2- 172.21.15.87:0/3661571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a4140610 0x7f75a41467b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.950+0000 7f75aa873640 1 -- 172.21.15.87:0/3661571470 >> 172.21.15.87:0/3661571470 conn(0x7f75a4068100 msgr2=0x7f75a406bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 1 -- 172.21.15.87:0/3661571470 shutdown_connections 2024-07-27T14:40:29.954 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 1 -- 172.21.15.87:0/3661571470 wait complete. 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 10 monclient: build_initial_monmap 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 10 monclient: monmap: 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:40:29.957 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 1 librados: starting msgr at 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.951+0000 7f75aa873640 1 librados: starting objecter 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 Processor -- start 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 -- start start 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 librados: setting wanted keys 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 librados: calling monclient init 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient: init 2024-07-27T14:40:29.958 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-c con 0x7f75a40aaa30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75a4140610 0x7f75a414b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4140610 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): picked mon.noname-b con 0x7f75a41463d0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f75a40b3fa0 con 0x7f75a4140610 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f75a40b3e20 con 0x7f75a41463d0 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:29.959 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f75a414d490 con 0x7f75a40aaa30 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.952+0000 7f75aa873640 10 monclient(hunting): authenticate will time out at 1169.978715s 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f759bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75a4140610 0x7f75a414b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a3fff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:46000/0 (socket says 172.21.15.87:46000) 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 1 -- 172.21.15.87:0/38295863 learned_addr learned my addr 172.21.15.87:0/38295863 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): get_auth_request con 0x7f75a41463d0 auth_method 0 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f759bfff640 10 monclient(hunting): get_auth_request con 0x7f75a4140610 auth_method 0 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f759bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:29.960 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f759bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f759bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75a40aaa30 auth_method 0 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.953+0000 7f75a8e10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f759bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:29.961 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.954+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 -- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 msgr2=0x7f75a40b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 -- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75a4140610 msgr2=0x7f75a414b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75a4140610 0x7f75a414b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 10 monclient: _finish_hunting 0 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 monclient: found mon.noname-b 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:29.962 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.956+0000 7f75a8e10640 1 -- 172.21.15.87:0/38295863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a414afd0 con 0x7f75a41463d0 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.958+0000 7f75a8e10640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f759c00efc0 tx=0x7f759c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.958+0000 7f75a17fa640 10 client.?.objecter ms_handle_connect 0x7f75a41463d0 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.958+0000 7f75a17fa640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c018020 con 0x7f75a41463d0 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: dump: 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:29.964 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: _finish_auth 0 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f759c014480 con 0x7f75a41463d0 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c014870 con 0x7f75a41463d0 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 5 monclient: authenticate success, global_id 4207 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a17fa640 10 monclient: dump: 2024-07-27T14:40:29.965 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 10 monclient: _renew_subs 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75a2ffd640 4 set_mon_vals no callback set 2024-07-27T14:40:29.966 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a414b2b0 con 0x7f75a41463d0 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 10 monclient: _renew_subs 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a414c330 con 0x7f75a41463d0 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.962+0000 7f75aa873640 1 librados: init done 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.963+0000 7f75aa873640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.963+0000 7f75aa873640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.963+0000 7f75aa873640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:29.967 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.963+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a40b32d0 con 0x7f75a41463d0 2024-07-27T14:40:29.982 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.981+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f759c005d10 con 0x7f75a41463d0 2024-07-27T14:40:29.982 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.981+0000 7f75a17fa640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 0x7f758406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:29.982 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.981+0000 7f75a3fff640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 0x7f758406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:29.982 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.981+0000 7f75a3fff640 10 monclient: get_auth_request con 0x7f758406d780 auth_method 0 2024-07-27T14:40:29.983 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.982+0000 7f75a3fff640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 0x7f758406fc40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7594007fd0 tx=0x7f7594007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:29.983 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.982+0000 7f75a17fa640 10 client.4207.objecter ms_handle_connect 0x7f758406d780 2024-07-27T14:40:29.983 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.881 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:29.984 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.984+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f759c0593e0 con 0x7f75a41463d0 2024-07-27T14:40:29.984 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.984+0000 7f75a17fa640 10 client.4207.objecter ms_dispatch 0x7f75a4000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:29.985 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.984+0000 7f75a17fa640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:29.985 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.984+0000 7f75a17fa640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:29.985 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:29.984+0000 7f75a17fa640 20 client.4207.objecter dump_active .. 0 homeless 2024-07-27T14:40:30.082 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.963 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.098 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:10.000 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.117 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:10.000 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.131 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.017 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.157 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.046 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.158 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:10.052 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.230 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.230+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f759c022030 con 0x7f75a41463d0 2024-07-27T14:40:30.230 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.230+0000 7f75a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:30.230 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.230+0000 7f75a17fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:30.244 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:10.131 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.252 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.149 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.346 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.227 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.363 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:10.265 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.389 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:10.273 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.407 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.406+0000 7f75aa873640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:30.407 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.406+0000 7f75aa873640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:30.407 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.406+0000 7f75aa873640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:30.407 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.406+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75a413f330 con 0x7f75a41463d0 2024-07-27T14:40:30.423 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.423+0000 7f75a17fa640 1 -- 172.21.15.87:0/38295863 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f759c010040 con 0x7f75a41463d0 2024-07-27T14:40:30.423 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.423+0000 7f75a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:30.423 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.423+0000 7f75a17fa640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:30.424 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:40:30.425 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:30.426 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:10.317 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 10 librados: watch_flush enter 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 10 librados: watch_flush exit 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 20 client.4207.objecter shutdown clearing up homeless session... 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 10 client.4207.objecter successfully canceled tick 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 msgr2=0x7f758406fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 0x7f758406fc40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7594007fd0 tx=0x7f7594007e80 comp rx=0 tx=0).stop 2024-07-27T14:40:30.427 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 10 monclient: shutdown 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 msgr2=0x7f75a414bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f759c00efc0 tx=0x7f759c00c490 comp rx=0 tx=0).stop 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 shutdown_connections 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f758406d780 0x7f758406fc40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75a41463d0 0x7f75a414bc10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75a4140610 0x7f75a414b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 --2- 172.21.15.87:0/38295863 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f75a40aaa30 0x7f75a40b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 >> 172.21.15.87:0/38295863 conn(0x7f75a4068100 msgr2=0x7f75a413da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 shutdown_connections 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 -- 172.21.15.87:0/38295863 wait complete. 2024-07-27T14:40:30.428 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:30.425+0000 7f75aa873640 1 librados: shutdown 2024-07-27T14:40:30.429 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.315 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.429 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.318 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-27T14:40:30.516 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.413 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.531 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:10.418 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.610 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.491 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.628 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:10.529 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.657 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:10.540 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.688 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:10.582 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.695 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.584 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.720 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.606 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.785 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.682 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.824 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:10.711 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.874 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.756 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.897 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:10.798 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.922 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:10.806 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:30.960 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.848 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.007 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.893 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.048 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.946 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.138 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.020 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.152 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:11.046 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.167 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:11.069 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.207 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:11.100 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.226 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:11.115 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.295 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:11.181 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.329 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.226 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.350 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:11.237 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.402 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:11.285 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.404 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:11.291 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.406 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.288 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.457 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:11.340 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.598 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:11.484 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.639 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:11.541 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.694 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:11.596 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.710 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:11.598 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.765 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:11.653 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.830 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.727 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.886 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.784 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:31.931 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.812 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:32.008 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.890 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:32.168 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:12.054 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:32.226 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:12.112 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:36.998 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:16.885 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.123 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:17.016 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.142 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:17.040 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.241 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:37.260 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:17.148 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.336 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.336+0000 7f1e73e31640 10 monclient: get_monmap_and_config 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.336+0000 7f1e73e31640 10 monclient: build_initial_monmap 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.336+0000 7f1e73e31640 10 monclient: monmap: 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:40:37.337471+0000 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:40:37.337471+0000 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:37.337 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 Processor -- start 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 -- start start 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient: init 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c1463d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 0x7f1e6c13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c0aaa30 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1e6c140440 0x7f1e6c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c140440 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:37.340 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6c0aae70 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6c0abe30 con 0x7f1e6c140440 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c146e50 con 0x7f1e6c1463d0 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.337+0000 7f1e73e31640 10 monclient(hunting): authenticate will time out at 1177.363977s 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 0x7f1e6c13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e71bcd640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e723ce640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1e6c140440 0x7f1e6c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 0x7f1e6c13ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36756/0 (socket says 172.21.15.87:36756) 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 1 -- 172.21.15.87:0/756324212 learned_addr learned my addr 172.21.15.87:0/756324212 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 10 monclient(hunting): get_auth_request con 0x7f1e6c0aaa30 auth_method 0 2024-07-27T14:40:37.341 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e713cc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e723ce640 10 monclient(hunting): get_auth_request con 0x7f1e6c140440 auth_method 0 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e723ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e723ce640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.338+0000 7f1e723ce640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): get_auth_request con 0x7f1e6c1463d0 auth_method 0 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.342 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.343 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.339+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 10 monclient(hunting): handle_auth_done global_id 4210 payload 274 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 -- 172.21.15.87:0/756324212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 msgr2=0x7f1e6c1467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 --2- 172.21.15.87:0/756324212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 -- 172.21.15.87:0/756324212 >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 msgr2=0x7f1e6c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 --2- 172.21.15.87:0/756324212 >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 0x7f1e6c13ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 10 monclient: _finish_hunting 0 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 monclient: found mon.noname-c 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.341+0000 7f1e723ce640 1 -- 172.21.15.87:0/756324212 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c0b3810 con 0x7f1e6c140440 2024-07-27T14:40:37.344 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.343+0000 7f1e723ce640 1 --2- 172.21.15.87:0/756324212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c140440 0x7f1e6c0b32d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0b2e10 tx=0x7f1e6402bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f6bc3999a8199 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 1 -- 172.21.15.87:0/756324212 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64039070 con 0x7f1e6c140440 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: dump: 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:37.348 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: _finish_auth 0 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: _check_auth_tickets 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 1 -- 172.21.15.87:0/756324212 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6402cb50 con 0x7f1e6c140440 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e73e31640 5 monclient: authenticate success, global_id 4210 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e73e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e73e31640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e73e31640 4 set_mon_vals no callback set 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 1 -- 172.21.15.87:0/756324212 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64034440 con 0x7f1e6c140440 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:37.349 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.347+0000 7f1e70bcb640 10 monclient: dump: 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:37.350 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.351 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:17.234 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 10 monclient: shutdown 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 -- 172.21.15.87:0/756324212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c140440 msgr2=0x7f1e6c0b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 --2- 172.21.15.87:0/756324212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c140440 0x7f1e6c0b32d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0b2e10 tx=0x7f1e6402bda0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 -- 172.21.15.87:0/756324212 shutdown_connections 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 --2- 172.21.15.87:0/756324212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c140440 0x7f1e6c0b32d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 --2- 172.21.15.87:0/756324212 >> v2:172.21.15.87:3300/0 conn(0x7f1e6c0aaa30 0x7f1e6c13ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.357 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 --2- 172.21.15.87:0/756324212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 -- 172.21.15.87:0/756324212 >> 172.21.15.87:0/756324212 conn(0x7f1e6c068100 msgr2=0x7f1e6c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.348+0000 7f1e73e31640 1 -- 172.21.15.87:0/756324212 shutdown_connections 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.349+0000 7f1e73e31640 1 -- 172.21.15.87:0/756324212 wait complete. 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 10 monclient: build_initial_monmap 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 10 monclient: monmap: 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:37.358 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 librados: starting msgr at 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 librados: starting objecter 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 Processor -- start 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 -- start start 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 librados: setting wanted keys 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 1 librados: calling monclient init 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.350+0000 7f1e73e31640 10 monclient: init 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:40:37.359 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c0aaa30 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e6c140440 0x7f1e6c1d8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c140440 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1d85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.351+0000 7f1e73e31640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c1463d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c0b3f20 con 0x7f1e6c140440 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6c0b3da0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:37.360 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6c142c60 con 0x7f1e6c1463d0 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e73e31640 10 monclient(hunting): authenticate will time out at 1177.378175s 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e713cc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e6c140440 0x7f1e6c1d8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:46846/0 (socket says 172.21.15.87:46846) 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 1 -- 172.21.15.87:0/1198477959 learned_addr learned my addr 172.21.15.87:0/1198477959 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 10 monclient(hunting): get_auth_request con 0x7f1e6c0aaa30 auth_method 0 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e71bcd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e723ce640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1d85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.361 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e713cc640 10 monclient(hunting): get_auth_request con 0x7f1e6c140440 auth_method 0 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e713cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e713cc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.352+0000 7f1e713cc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): get_auth_request con 0x7f1e6c1463d0 auth_method 0 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.362 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e713cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.353+0000 7f1e723ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 10 monclient(hunting): handle_auth_done global_id 4213 payload 824 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 -- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 msgr2=0x7f1e6c1d85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1d85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 -- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e6c140440 msgr2=0x7f1e6c1d8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e6c140440 0x7f1e6c1d8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 10 monclient: _finish_hunting 0 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 monclient: found mon.noname-b 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.363 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.355+0000 7f1e71bcd640 1 -- 172.21.15.87:0/1198477959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6c0ac810 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.365 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.358+0000 7f1e627fc640 10 client.?.objecter ms_handle_connect 0x7f1e6c0aaa30 2024-07-27T14:40:37.365 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.358+0000 7f1e627fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.358+0000 7f1e71bcd640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e8a0 tx=0x7f1e6800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68017070 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 10 monclient: dump: 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.366 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 10 monclient: _finish_auth 0 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 10 monclient: _check_auth_tickets 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 5 monclient: authenticate success, global_id 4213 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 10 monclient: _renew_subs 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6c0acaf0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 10 monclient: _renew_subs 2024-07-27T14:40:37.367 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6c1d8ce0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.360+0000 7f1e73e31640 1 librados: init done 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.361+0000 7f1e73e31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.361+0000 7f1e73e31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.361+0000 7f1e73e31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.361+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6c0accc0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6800ee90 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68021060 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.368 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e627fc640 10 monclient: dump: 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:37.369 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:37.370 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:37.370 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.362+0000 7f1e63fff640 4 set_mon_vals no callback set 2024-07-27T14:40:37.424 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:17.307 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.425 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:17.328 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.589 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:17.471 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:37.635 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.635+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f1e6801a070 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.636 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.636+0000 7f1e627fc640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 0x7f1e5006fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:37.636 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.636+0000 7f1e713cc640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 0x7f1e5006fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:37.636 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.636+0000 7f1e713cc640 10 monclient: get_auth_request con 0x7f1e5006d730 auth_method 0 2024-07-27T14:40:37.637 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.637+0000 7f1e627fc640 10 client.4213.objecter ms_handle_connect 0x7f1e5006d730 2024-07-27T14:40:37.637 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.637+0000 7f1e713cc640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 0x7f1e5006fbf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c1dc450 tx=0x7f1e5c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:37.639 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.639+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1e680898e0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.639 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.639+0000 7f1e627fc640 10 client.4213.objecter ms_dispatch 0x7f1e6c000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:37.639 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.639+0000 7f1e627fc640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:37.640 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.639+0000 7f1e627fc640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:37.640 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.639+0000 7f1e627fc640 20 client.4213.objecter dump_active .. 0 homeless 2024-07-27T14:40:37.652 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.652+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f1e680596e0 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.653 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.652+0000 7f1e627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:37.653 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.652+0000 7f1e627fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:37.871 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.870+0000 7f1e73e31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:37.871 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.870+0000 7f1e73e31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:37.871 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.870+0000 7f1e73e31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:40:37.871 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.870+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e6c13ed80 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.890+0000 7f1e627fc640 1 -- 172.21.15.87:0/1198477959 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1e6c13ed80 con 0x7f1e6c0aaa30 2024-07-27T14:40:37.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.890+0000 7f1e627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:37.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.890+0000 7f1e627fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:37.891 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:40:37.891 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 10 librados: watch_flush enter 2024-07-27T14:40:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 10 librados: watch_flush exit 2024-07-27T14:40:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 20 client.4213.objecter shutdown clearing up homeless session... 2024-07-27T14:40:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 10 client.4213.objecter successfully canceled tick 2024-07-27T14:40:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 msgr2=0x7f1e5006fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 0x7f1e5006fbf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c1dc450 tx=0x7f1e5c009240 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 10 monclient: shutdown 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 msgr2=0x7f1e6c1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e8a0 tx=0x7f1e6800eca0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 shutdown_connections 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e5006d730 0x7f1e5006fbf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e6c1463d0 0x7f1e6c1d85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e6c140440 0x7f1e6c1d8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 --2- 172.21.15.87:0/1198477959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e6c0aaa30 0x7f1e6c1cf020 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.892+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 >> 172.21.15.87:0/1198477959 conn(0x7f1e6c068100 msgr2=0x7f1e6c0b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:37.896 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.895+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 shutdown_connections 2024-07-27T14:40:37.896 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.895+0000 7f1e73e31640 1 -- 172.21.15.87:0/1198477959 wait complete. 2024-07-27T14:40:37.896 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:37.895+0000 7f1e73e31640 1 librados: shutdown 2024-07-27T14:40:37.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-27T14:40:42.819 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:22.720 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:42.822 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:22.725 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:42.969 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:22.856 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:42.973 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:22.860 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:42.994 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:22.882 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:42.998 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:22.886 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.031 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:22.914 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.035 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:22.918 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.045 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:22.939 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.052 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:22.945 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.146 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:23.044 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.151 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:23.049 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.182 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:23.068 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.187 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:23.074 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.354 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:23.235 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:43.360 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:23.242 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:44.715 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:44.834 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.834+0000 7ff4343b2640 10 monclient: get_monmap_and_config 2024-07-27T14:40:44.834 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.834+0000 7ff4343b2640 10 monclient: build_initial_monmap 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.834+0000 7ff4343b2640 10 monclient: monmap: 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:40:44.835477+0000 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:40:44.835477+0000 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:44.835 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 Processor -- start 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 -- start start 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient: init 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 0x7ff42c1467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-b con 0x7ff42c1463d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:44.837 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff42c0aaa30 addr v1:172.21.15.112:6789/0 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff42c140610 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42c0aae70 con 0x7ff42c140610 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42c0abe30 con 0x7ff42c0aaa30 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff42c146e50 con 0x7ff42c1463d0 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.835+0000 7ff4343b2640 10 monclient(hunting): authenticate will time out at 1184.861986s 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.836+0000 7ff43214e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 0x7ff42c1467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.836+0000 7ff43194d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff42c0aaa30 0x7ff42c13ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:60488/0 (socket says 172.21.15.87:60488) 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.836+0000 7ff43194d640 1 -- 172.21.15.87:0/1716977078 learned_addr learned my addr 172.21.15.87:0/1716977078 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): get_auth_request con 0x7ff42c1463d0 auth_method 0 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:44.838 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:44.839 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:44.839 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:44.839 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.837+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 1 -- 172.21.15.87:0/1716977078 >> v1:172.21.15.112:6789/0 conn(0x7ff42c0aaa30 legacy=0x7ff42c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 1 -- 172.21.15.87:0/1716977078 >> v1:172.21.15.87:6789/0 conn(0x7ff42c140610 legacy=0x7ff42c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 10 monclient: _finish_hunting 0 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 1 monclient: found mon.noname-b 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:40:44.840 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.839+0000 7ff43214e640 1 -- 172.21.15.87:0/1716977078 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c0b39e0 con 0x7ff42c1463d0 2024-07-27T14:40:44.841 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.841+0000 7ff43214e640 1 --2- 172.21.15.87:0/1716977078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 0x7ff42c1467b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff428009800 tx=0x7ff42802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cbfcb35946ed5c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:44.846 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 1 -- 172.21.15.87:0/1716977078 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428035020 con 0x7ff42c1463d0 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: dump: 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.847 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: _finish_auth 0 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: _check_auth_tickets 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 1 -- 172.21.15.87:0/1716977078 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff428031800 con 0x7ff42c1463d0 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 5 monclient: authenticate success, global_id 4193 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 1 -- 172.21.15.87:0/1716977078 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428031bd0 con 0x7ff42c1463d0 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff43114c640 10 monclient: dump: 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.848 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 4 set_mon_vals no callback set 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 10 monclient: shutdown 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 1 -- 172.21.15.87:0/1716977078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 msgr2=0x7ff42c1467b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.845+0000 7ff4343b2640 1 --2- 172.21.15.87:0/1716977078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 0x7ff42c1467b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff428009800 tx=0x7ff42802b8a0 comp rx=0 tx=0).stop 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.846+0000 7ff4343b2640 1 -- 172.21.15.87:0/1716977078 shutdown_connections 2024-07-27T14:40:44.849 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.846+0000 7ff4343b2640 1 --2- 172.21.15.87:0/1716977078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c1463d0 0x7ff42c1467b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:44.850 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.846+0000 7ff4343b2640 1 -- 172.21.15.87:0/1716977078 >> 172.21.15.87:0/1716977078 conn(0x7ff42c068100 msgr2=0x7ff42c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:44.850 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.846+0000 7ff4343b2640 1 -- 172.21.15.87:0/1716977078 shutdown_connections 2024-07-27T14:40:44.850 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.846+0000 7ff4343b2640 1 -- 172.21.15.87:0/1716977078 wait complete. 2024-07-27T14:40:44.855 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 10 monclient: build_initial_monmap 2024-07-27T14:40:44.855 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 10 monclient: monmap: 2024-07-27T14:40:44.855 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:44.855 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:44.855 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 librados: starting msgr at 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 librados: starting objecter 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 Processor -- start 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 -- start start 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 librados: setting wanted keys 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 1 librados: calling monclient init 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.847+0000 7ff4343b2640 10 monclient: init 2024-07-27T14:40:44.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-c con 0x7ff42c0aaa30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff42c140610 0x7ff42c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-b con 0x7ff42c140610 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff42c1463d0 0x7ff42c13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): picked mon.noname-a con 0x7ff42c1463d0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c14e4a0 con 0x7ff42c1463d0 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c14e620 con 0x7ff42c140610 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:44.857 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff42c14e7a0 con 0x7ff42c0aaa30 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.848+0000 7ff4343b2640 10 monclient(hunting): authenticate will time out at 1184.874914s 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43194d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff42c140610 0x7ff42c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43294f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff42c1463d0 0x7ff42c13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43666/0 (socket says 172.21.15.87:43666) 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 1 -- 172.21.15.87:0/4154177082 learned_addr learned my addr 172.21.15.87:0/4154177082 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 10 monclient(hunting): get_auth_request con 0x7ff42c0aaa30 auth_method 0 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43214e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43194d640 10 monclient(hunting): get_auth_request con 0x7ff42c140610 auth_method 0 2024-07-27T14:40:44.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43194d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43194d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43194d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43294f640 10 monclient(hunting): get_auth_request con 0x7ff42c1463d0 auth_method 0 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43294f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43294f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.849+0000 7ff43294f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43194d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43194d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43194d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43294f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43294f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43294f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:44.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.850+0000 7ff43214e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 -- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff42c140610 msgr2=0x7ff42c13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff42c140610 0x7ff42c13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 -- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff42c1463d0 msgr2=0x7ff42c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff42c1463d0 0x7ff42c13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 10 monclient: _finish_hunting 0 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 monclient: found mon.noname-c 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:40:44.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.853+0000 7ff43214e640 1 -- 172.21.15.87:0/4154177082 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c071600 con 0x7ff42c0aaa30 2024-07-27T14:40:44.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.855+0000 7ff43214e640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff428009920 tx=0x7ff4280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:44.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.855+0000 7ff422ffd640 10 client.?.objecter ms_handle_connect 0x7ff42c0aaa30 2024-07-27T14:40:44.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.855+0000 7ff422ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:44.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428041070 con 0x7ff42c0aaa30 2024-07-27T14:40:44.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: dump: 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: _finish_auth 0 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42803f070 con 0x7ff42c0aaa30 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff422ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.860+0000 7ff43094b640 4 set_mon_vals no callback set 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428035040 con 0x7ff42c0aaa30 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff422ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff422ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff422ffd640 10 monclient: dump: 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:44.863 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 5 monclient: authenticate success, global_id 4196 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 10 monclient: _renew_subs 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42c13bb00 con 0x7ff42c0aaa30 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 10 monclient: _renew_subs 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42c06fa80 con 0x7ff42c0aaa30 2024-07-27T14:40:44.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.861+0000 7ff4343b2640 1 librados: init done 2024-07-27T14:40:44.865 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.862+0000 7ff4343b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:44.865 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.862+0000 7ff4343b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:44.865 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.862+0000 7ff4343b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:44.865 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.862+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff42c1467b0 con 0x7ff42c0aaa30 2024-07-27T14:40:44.889 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.888+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7ff428038020 con 0x7ff42c0aaa30 2024-07-27T14:40:44.889 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.889+0000 7ff422ffd640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 0x7ff41006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:44.889 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.889+0000 7ff43194d640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 0x7ff41006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:44.889 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.889+0000 7ff43194d640 10 monclient: get_auth_request con 0x7ff41006d780 auth_method 0 2024-07-27T14:40:44.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.890+0000 7ff43194d640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 0x7ff41006fc40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff42c06e370 tx=0x7ff41c007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:44.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.890+0000 7ff422ffd640 10 client.4196.objecter ms_handle_connect 0x7ff41006d780 2024-07-27T14:40:44.893 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.893+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff428079dd0 con 0x7ff42c0aaa30 2024-07-27T14:40:44.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.893+0000 7ff422ffd640 10 client.4196.objecter ms_dispatch 0x7ff42c000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:44.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.893+0000 7ff422ffd640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:44.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.893+0000 7ff422ffd640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:44.894 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:44.893+0000 7ff422ffd640 20 client.4196.objecter dump_active .. 0 homeless 2024-07-27T14:40:45.219 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.219+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7ff42807b140 con 0x7ff42c0aaa30 2024-07-27T14:40:45.219 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.219+0000 7ff422ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:45.219 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.219+0000 7ff422ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:45.529 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.528+0000 7ff4343b2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:45.529 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.528+0000 7ff4343b2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:45.529 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.528+0000 7ff4343b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:45.529 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.528+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff42c071cc0 con 0x7ff42c0aaa30 2024-07-27T14:40:45.569 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.569+0000 7ff422ffd640 1 -- 172.21.15.87:0/4154177082 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff42807f440 con 0x7ff42c0aaa30 2024-07-27T14:40:45.570 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.569+0000 7ff422ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:45.570 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:40:45.570 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.569+0000 7ff422ffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:45.570 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.571+0000 7ff4343b2640 10 librados: watch_flush enter 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.571+0000 7ff4343b2640 10 librados: watch_flush exit 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.571+0000 7ff4343b2640 20 client.4196.objecter shutdown clearing up homeless session... 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.571+0000 7ff4343b2640 10 client.4196.objecter successfully canceled tick 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 msgr2=0x7ff41006fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 0x7ff41006fc40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff42c06e370 tx=0x7ff41c007e80 comp rx=0 tx=0).stop 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 10 monclient: shutdown 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 msgr2=0x7ff42c06f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:45.572 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff428009920 tx=0x7ff4280046c0 comp rx=0 tx=0).stop 2024-07-27T14:40:45.573 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 shutdown_connections 2024-07-27T14:40:45.573 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff41006d780 0x7ff41006fc40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:45.573 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff42c1463d0 0x7ff42c13b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:45.573 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff42c140610 0x7ff42c13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:45.573 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 --2- 172.21.15.87:0/4154177082 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff42c0aaa30 0x7ff42c06f360 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:45.574 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 >> 172.21.15.87:0/4154177082 conn(0x7ff42c068100 msgr2=0x7ff42c13da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:45.574 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 shutdown_connections 2024-07-27T14:40:45.574 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 -- 172.21.15.87:0/4154177082 wait complete. 2024-07-27T14:40:45.574 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:45.572+0000 7ff4343b2640 1 librados: shutdown 2024-07-27T14:40:45.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-27T14:40:47.135 INFO:tasks.ceph.osd.4.smithi112.stderr:==00:00:00:27.037 35806== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.170 INFO:tasks.ceph.osd.7.smithi112.stderr:==00:00:00:27.053 35815== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.333 INFO:tasks.ceph.osd.5.smithi112.stderr:==00:00:00:27.226 35807== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.378 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:27.267 35684== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.517 INFO:tasks.ceph.osd.6.smithi112.stderr:==00:00:00:27.404 35814== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.775 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:27.661 35685== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.852 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:27.733 35686== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:47.852 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:27.750 35683== Warning: unimplemented fcntl command: 1038 2024-07-27T14:40:52.396 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:40:52.532 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.532+0000 7ffab7d84640 10 monclient: get_monmap_and_config 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.532+0000 7ffab7d84640 10 monclient: build_initial_monmap 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.532+0000 7ffab7d84640 10 monclient: monmap: 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:40:52.533405+0000 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:40:52.533405+0000 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:40:52.533 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:40:52.534 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:40:52.534 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:40:52.534 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 Processor -- start 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 -- start start 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient: init 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffab01463d0 addr v1:172.21.15.87:6789/0 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ffab0140610 0x7ffab01467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-c con 0x7ffab0140610 addr v2:172.21.15.112:3300/0 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ffab00aaa30 0x7ffab00b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-a con 0x7ffab00aaa30 addr v2:172.21.15.87:3300/0 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab00abe30 con 0x7ffab01463d0 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:52.536 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7ffab00aae80 con 0x7ffab00aaa30 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7ffab0146e50 con 0x7ffab0140610 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.533+0000 7ffab7d84640 10 monclient(hunting): authenticate will time out at 1192.559959s 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ffab0140610 0x7ffab01467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ffab0140610 0x7ffab01467b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:41096/0 (socket says 172.21.15.87:41096) 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 1 -- 172.21.15.87:0/3799180735 learned_addr learned my addr 172.21.15.87:0/3799180735 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 10 monclient(hunting): get_auth_request con 0x7ffab0140610 auth_method 0 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.534+0000 7ffab531f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.537 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.535+0000 7ffab6321640 1 --2- 172.21.15.87:0/3799180735 >> v2:172.21.15.87:3300/0 conn(0x7ffab00aaa30 0x7ffab00b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.538 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.535+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:52.538 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.535+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:52.538 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.535+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:52.539 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.536+0000 7ffab6321640 10 monclient(hunting): get_auth_request con 0x7ffab00aaa30 auth_method 0 2024-07-27T14:40:52.539 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.536+0000 7ffab6321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:52.539 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.536+0000 7ffab6321640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.539 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.536+0000 7ffab6321640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 1 -- 172.21.15.87:0/3799180735 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1513550441 0 0) 0x7ffab00abe30 con 0x7ffab01463d0 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 10 monclient(hunting): _negotiate 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 10 monclient(hunting): my global_id is 4206 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab4b1e640 1 -- 172.21.15.87:0/3799180735 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa9c003520 con 0x7ffab01463d0 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 1 -- 172.21.15.87:0/3799180735 >> v2:172.21.15.87:3300/0 conn(0x7ffab00aaa30 msgr2=0x7ffab00b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 1 --2- 172.21.15.87:0/3799180735 >> v2:172.21.15.87:3300/0 conn(0x7ffab00aaa30 0x7ffab00b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 1 -- 172.21.15.87:0/3799180735 >> v1:172.21.15.87:6789/0 conn(0x7ffab01463d0 legacy=0x7ffab013ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 10 monclient: _finish_hunting 0 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 1 monclient: found mon.noname-c 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:40:52.540 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.538+0000 7ffab531f640 1 -- 172.21.15.87:0/3799180735 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab00b3810 con 0x7ffab0140610 2024-07-27T14:40:52.541 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.541+0000 7ffab531f640 1 --2- 172.21.15.87:0/3799180735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab0140610 0x7ffab01467b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffab0072450 tx=0x7ffaa002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326345275d15eae7 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 1 -- 172.21.15.87:0/3799180735 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0035020 con 0x7ffab0140610 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: dump: 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.547 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: _finish_auth 0 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: _check_auth_tickets 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 1 -- 172.21.15.87:0/3799180735 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa0031870 con 0x7ffab0140610 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:52.548 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 1 -- 172.21.15.87:0/3799180735 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0031c40 con 0x7ffab0140610 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.546+0000 7ffab4b1e640 10 monclient: dump: 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:52.549 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:52.550 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.550 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 5 monclient: authenticate success, global_id 4222 2024-07-27T14:40:52.550 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 10 monclient: get_monmap_and_config success 2024-07-27T14:40:52.550 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 4 set_mon_vals no callback set 2024-07-27T14:40:52.555 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 10 monclient: shutdown 2024-07-27T14:40:52.555 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 -- 172.21.15.87:0/3799180735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab0140610 msgr2=0x7ffab01467b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 --2- 172.21.15.87:0/3799180735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab0140610 0x7ffab01467b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffab0072450 tx=0x7ffaa002b910 comp rx=0 tx=0).stop 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 -- 172.21.15.87:0/3799180735 shutdown_connections 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 --2- 172.21.15.87:0/3799180735 >> v2:172.21.15.87:3300/0 conn(0x7ffab00aaa30 0x7ffab00b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 --2- 172.21.15.87:0/3799180735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab0140610 0x7ffab01467b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 -- 172.21.15.87:0/3799180735 >> 172.21.15.87:0/3799180735 conn(0x7ffab0068100 msgr2=0x7ffab006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.547+0000 7ffab7d84640 1 -- 172.21.15.87:0/3799180735 shutdown_connections 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.548+0000 7ffab7d84640 1 -- 172.21.15.87:0/3799180735 wait complete. 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.548+0000 7ffab7d84640 10 monclient: build_initial_monmap 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.548+0000 7ffab7d84640 10 monclient: monmap: 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:40:52.556 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.548+0000 7ffab7d84640 1 librados: starting msgr at 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.548+0000 7ffab7d84640 1 librados: starting objecter 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 Processor -- start 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 -- start start 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 librados: setting wanted keys 2024-07-27T14:40:52.557 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 librados: calling monclient init 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient: init 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient: _reopen_session rank -1 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 0x7ffab013d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-b con 0x7ffab00aaa30 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 0x7ffab013bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-a con 0x7ffab0140610 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 0x7ffab013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): picked mon.noname-c con 0x7ffab01463d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffab00b3f60 con 0x7ffab0140610 2024-07-27T14:40:52.558 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffab00b3de0 con 0x7ffab00aaa30 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): start opening mon connection 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ffab014d490 con 0x7ffab01463d0 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): _renew_subs 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.549+0000 7ffab7d84640 10 monclient(hunting): authenticate will time out at 1192.575674s 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab531f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 0x7ffab013bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab6321640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 0x7ffab013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 0x7ffab013d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab531f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 0x7ffab013bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43208/0 (socket says 172.21.15.87:43208) 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 0x7ffab013d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:41098/0 (socket says 172.21.15.87:41098) 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 1 -- 172.21.15.87:0/2796014214 learned_addr learned my addr 172.21.15.87:0/2796014214 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 10 monclient(hunting): get_auth_request con 0x7ffab00aaa30 auth_method 0 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:52.559 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab5b20640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab6321640 10 monclient(hunting): get_auth_request con 0x7ffab01463d0 auth_method 0 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab6321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab6321640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.550+0000 7ffab6321640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab531f640 10 monclient(hunting): get_auth_request con 0x7ffab0140610 auth_method 0 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab531f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab531f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab531f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab6321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab6321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab6321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab5b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:52.560 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab5b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab5b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.551+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.552+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.552+0000 7ffab531f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 -- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 msgr2=0x7ffab013d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 0x7ffab013d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 -- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 msgr2=0x7ffab013bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 0x7ffab013bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 10 monclient: _finish_hunting 0 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 monclient: found mon.noname-c 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:40:52.561 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:40:52.562 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.554+0000 7ffab6321640 1 -- 172.21.15.87:0/2796014214 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab01414a0 con 0x7ffab01463d0 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.556+0000 7ffab6321640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 0x7ffab013dec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8007d20 tx=0x7ffaa800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.556+0000 7ffaa67fc640 10 client.?.objecter ms_handle_connect 0x7ffab01463d0 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.556+0000 7ffaa67fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800ed70 con 0x7ffab01463d0 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: dump: 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.565 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: _finish_auth 0 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: _check_auth_tickets 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:40:52.566 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa8003990 con 0x7ffab01463d0 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 5 monclient: authenticate success, global_id 4199 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 10 monclient: _renew_subs 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab013c450 con 0x7ffab01463d0 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8003d80 con 0x7ffab01463d0 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa67fc640 10 monclient: dump: 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:40:52.567 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffaa7fff640 4 set_mon_vals no callback set 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 10 monclient: _renew_subs 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab013c8b0 con 0x7ffab01463d0 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.561+0000 7ffab7d84640 1 librados: init done 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.562+0000 7ffab7d84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:52.568 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.562+0000 7ffab7d84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:52.569 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.562+0000 7ffab7d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:52.569 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.562+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab00b32d0 con 0x7ffab01463d0 2024-07-27T14:40:52.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.585+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7ffaa801b020 con 0x7ffab01463d0 2024-07-27T14:40:52.585 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.585+0000 7ffaa67fc640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 0x7ffa9406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:40:52.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.586+0000 7ffab5b20640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 0x7ffa9406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:40:52.586 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.586+0000 7ffab5b20640 10 monclient: get_auth_request con 0x7ffa9406d780 auth_method 0 2024-07-27T14:40:52.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.586+0000 7ffaa67fc640 10 client.4199.objecter ms_handle_connect 0x7ffa9406d780 2024-07-27T14:40:52.587 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.586+0000 7ffab5b20640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 0x7ffa9406fc40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffaac007fd0 tx=0x7ffaac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:40:52.897 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ffaa80589b0 con 0x7ffab01463d0 2024-07-27T14:40:52.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 10 client.4199.objecter ms_dispatch 0x7ffab0000b90 osd_map(10..10 src has 1..10) 2024-07-27T14:40:52.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 3 client.4199.objecter handle_osd_map got epochs [10,10] > 0 2024-07-27T14:40:52.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 3 client.4199.objecter handle_osd_map decoding full epoch 10 2024-07-27T14:40:52.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 20 client.4199.objecter dump_active .. 0 homeless 2024-07-27T14:40:52.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7ffaa8059d20 con 0x7ffab01463d0 2024-07-27T14:40:52.899 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:40:52.899 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:52.897+0000 7ffaa67fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:40:53.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.235+0000 7ffab7d84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:53.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.235+0000 7ffab7d84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:53.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.235+0000 7ffab7d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:40:53.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.235+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffab00b6900 con 0x7ffab01463d0 2024-07-27T14:40:53.261 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.261+0000 7ffaa67fc640 1 -- 172.21.15.87:0/2796014214 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ffaa8060020 con 0x7ffab01463d0 2024-07-27T14:40:53.262 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.261+0000 7ffaa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:40:53.262 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.261+0000 7ffaa67fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:40:53.262 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:40:53.262 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:19.591859+0000","last_up_change":"0.000000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 10 librados: watch_flush enter 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 10 librados: watch_flush exit 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 20 client.4199.objecter shutdown clearing up homeless session... 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 10 client.4199.objecter successfully canceled tick 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 msgr2=0x7ffa9406fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 0x7ffa9406fc40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffaac007fd0 tx=0x7ffaac009040 comp rx=0 tx=0).stop 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 10 monclient: shutdown 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:40:53.266 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 msgr2=0x7ffab013dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 0x7ffab013dec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8007d20 tx=0x7ffaa800e4b0 comp rx=0 tx=0).stop 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 shutdown_connections 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ffa9406d780 0x7ffa9406fc40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffab01463d0 0x7ffab013dec0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffab0140610 0x7ffab013bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 --2- 172.21.15.87:0/2796014214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffab00aaa30 0x7ffab013d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.264+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 >> 172.21.15.87:0/2796014214 conn(0x7ffab0068100 msgr2=0x7ffab00b4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.265+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 shutdown_connections 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.265+0000 7ffab7d84640 1 -- 172.21.15.87:0/2796014214 wait complete. 2024-07-27T14:40:53.267 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:40:53.265+0000 7ffab7d84640 1 librados: shutdown 2024-07-27T14:40:53.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-27T14:40:55.492 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:40:55.481+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-07-27T14:40:55.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:40:55.747+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-07-27T14:40:55.789 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:40:55.784+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-07-27T14:40:55.805 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:40:55.800+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-07-27T14:40:56.140 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:40:56.134+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-07-27T14:40:56.189 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:40:56.181+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-07-27T14:40:56.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:40:56.343+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-07-27T14:40:56.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:40:56.357+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-07-27T14:40:58.584 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:40:58.567+0000 c49a640 -1 osd.4 0 waiting for initial osdmap 2024-07-27T14:40:58.621 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:40:58.607+0000 c49a640 -1 osd.5 0 waiting for initial osdmap 2024-07-27T14:40:58.633 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:40:58.629+0000 c49a640 -1 osd.7 0 waiting for initial osdmap 2024-07-27T14:40:59.023 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:40:59.009+0000 10260640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:40:59.025 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:40:59.011+0000 10260640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:40:59.055 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:40:59.041+0000 10260640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:40:59.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:40:59.783+0000 c49a640 -1 osd.0 0 waiting for initial osdmap 2024-07-27T14:40:59.806 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:40:59.803+0000 c49a640 -1 osd.3 0 waiting for initial osdmap 2024-07-27T14:40:59.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:40:59.813+0000 ac84640 -1 osd.1 0 waiting for initial osdmap 2024-07-27T14:41:00.086 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:41:00.211 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.210+0000 7f9cd1c5e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.210+0000 7f9cd1c5e640 10 monclient: build_initial_monmap 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.210+0000 7f9cd1c5e640 10 monclient: monmap: 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:41:00.211933+0000 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:41:00.211933+0000 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:00.212 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:00.213 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:00.213 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.211+0000 7f9cd1c5e640 1 Processor -- start 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.212+0000 7f9cd1c5e640 1 -- start start 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.212+0000 7f9cd1c5e640 10 monclient: init 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.212+0000 7f9cd1c5e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.212+0000 7f9cd1c5e640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.214 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ccc0b92d0 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ccc0b2d80 0x7f9ccc0aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc0b2d80 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ccc140050 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc0b5230 con 0x7f9ccc140050 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f9ccc0b96b0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f9ccc0b4640 con 0x7f9ccc0b2d80 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9cd1c5e640 10 monclient(hunting): authenticate will time out at 1200.239287s 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9ccb7fe640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.215 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.213+0000 7f9ccaffd640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ccc0b2d80 0x7f9ccc0aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.216 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35734/0 (socket says 172.21.15.87:35734) 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 1 -- 172.21.15.87:0/198817800 learned_addr learned my addr 172.21.15.87:0/198817800 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b92d0 auth_method 0 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b2d80 auth_method 0 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccaffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.214+0000 7f9ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.215+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:00.217 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.215+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.215+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 1 -- 172.21.15.87:0/198817800 >> v2:172.21.15.112:3300/0 conn(0x7f9ccc0b2d80 msgr2=0x7f9ccc0aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 1 --2- 172.21.15.87:0/198817800 >> v2:172.21.15.112:3300/0 conn(0x7f9ccc0b2d80 0x7f9ccc0aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 1 -- 172.21.15.87:0/198817800 >> v1:172.21.15.112:6789/0 conn(0x7f9ccc140050 legacy=0x7f9ccc1460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 1 monclient: found mon.noname-a 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.218 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.216+0000 7f9ccb7fe640 1 -- 172.21.15.87:0/198817800 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc1467c0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.219 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.219+0000 7f9ccb7fe640 1 --2- 172.21.15.87:0/198817800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc009870 tx=0x7f9cbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09b74ed19c8dc72 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:00.225 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 1 -- 172.21.15.87:0/198817800 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc02e070 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.225 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: dump: 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.226 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 1 -- 172.21.15.87:0/198817800 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc037070 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 1 -- 172.21.15.87:0/198817800 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc031a00 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cca7fc640 10 monclient: dump: 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.227 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cd1c5e640 5 monclient: authenticate success, global_id 4230 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cd1c5e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:00.228 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.225+0000 7f9cd1c5e640 4 set_mon_vals no callback set 2024-07-27T14:41:00.231 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 10 monclient: shutdown 2024-07-27T14:41:00.231 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:00.231 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/198817800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 msgr2=0x7f9ccc0b4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/198817800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc009870 tx=0x7f9cbc02b880 comp rx=0 tx=0).stop 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/198817800 shutdown_connections 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/198817800 >> v2:172.21.15.112:3300/0 conn(0x7f9ccc0b2d80 0x7f9ccc0aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/198817800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc0b4b60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/198817800 >> 172.21.15.87:0/198817800 conn(0x7f9ccc067d80 msgr2=0x7f9ccc0b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/198817800 shutdown_connections 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.226+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/198817800 wait complete. 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.227+0000 7f9cd1c5e640 10 monclient: build_initial_monmap 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.227+0000 7f9cd1c5e640 10 monclient: monmap: 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:41:00.232 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 librados: starting msgr at 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 librados: starting objecter 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 Processor -- start 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 -- start start 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 librados: setting wanted keys 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 1 librados: calling monclient init 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.228+0000 7f9cd1c5e640 10 monclient: init 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:00.233 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ccc0b2d80 0x7f9ccc14a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc0b2d80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ccc0b92d0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ccc140050 0x7f9ccc13cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ccc140050 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc14d3e0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc1406d0 con 0x7f9ccc140050 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:00.234 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f9ccc14d560 con 0x7f9ccc0b2d80 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9cd1c5e640 10 monclient(hunting): authenticate will time out at 1200.255427s 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ccc140050 0x7f9ccc13cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.229+0000 7f9ccaffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccb7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ccc0b2d80 0x7f9ccc14a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35750/0 (socket says 172.21.15.87:35750) 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 1 -- 172.21.15.87:0/582085016 learned_addr learned my addr 172.21.15.87:0/582085016 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b92d0 auth_method 0 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request con 0x7f9ccc140050 auth_method 0 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:00.235 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b2d80 auth_method 0 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.230+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:00.236 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.231+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:00.237 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-07-27T14:41:00.237 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 -- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ccc0b2d80 msgr2=0x7f9ccc14a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.237 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ccc0b2d80 0x7f9ccc14a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 -- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ccc140050 msgr2=0x7f9ccc13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ccc140050 0x7f9ccc13cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 monclient: found mon.noname-a 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.233+0000 7f9ccaffd640 1 -- 172.21.15.87:0/582085016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc14bda0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.236+0000 7f9ccaffd640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000e930 tx=0x7f9cc000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.236+0000 7f9cb3fff640 10 client.?.objecter ms_handle_connect 0x7f9ccc0b92d0 2024-07-27T14:41:00.238 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.236+0000 7f9cb3fff640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0017070 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: dump: 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:00.242 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc000c9a0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc000cd50 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cb3fff640 10 monclient: dump: 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:00.243 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cd1c5e640 5 monclient: authenticate success, global_id 4233 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 10 monclient: _renew_subs 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ccc14c080 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.240+0000 7f9cc9ffb640 4 set_mon_vals no callback set 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:00.244 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 10 monclient: _renew_subs 2024-07-27T14:41:00.245 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.245 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ccc14afd0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.245 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.241+0000 7f9cd1c5e640 1 librados: init done 2024-07-27T14:41:00.245 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.242+0000 7f9cd1c5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:00.245 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.242+0000 7f9cd1c5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:00.246 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.242+0000 7f9cd1c5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.246 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.242+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ccc0aaa10 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.277 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.277+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f9cc001a070 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.278 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.277+0000 7f9cb3fff640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 0x7f9cb806fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:00.278 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.278+0000 7f9ccb7fe640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 0x7f9cb806fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:00.278 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.278+0000 7f9ccb7fe640 10 monclient: get_auth_request con 0x7f9cb806d760 auth_method 0 2024-07-27T14:41:00.278 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.278+0000 7f9cb3fff640 10 client.4233.objecter ms_handle_connect 0x7f9cb806d760 2024-07-27T14:41:00.278 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.278+0000 7f9ccb7fe640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 0x7f9cb806fc20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc036000 tx=0x7f9cbc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:00.283 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.283+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3071+0+0 (secure 0 0 0) 0x7f9cc0057a50 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.284 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.283+0000 7f9cb3fff640 10 client.4233.objecter ms_dispatch 0x7f9ccc000b90 osd_map(13..13 src has 1..13) 2024-07-27T14:41:00.284 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.283+0000 7f9cb3fff640 3 client.4233.objecter handle_osd_map got epochs [13,13] > 0 2024-07-27T14:41:00.284 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.283+0000 7f9cb3fff640 3 client.4233.objecter handle_osd_map decoding full epoch 13 2024-07-27T14:41:00.284 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.283+0000 7f9cb3fff640 20 client.4233.objecter dump_active .. 0 homeless 2024-07-27T14:41:00.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:41:00.272+0000 10260640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:41:00.289 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:41:00.272+0000 10260640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:41:00.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:41:00.274+0000 10260640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:41:00.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.632+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f9cc00599b0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.633 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.632+0000 7f9cb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:00.633 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.632+0000 7f9cb3fff640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:00.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.866+0000 7f9cd1c5e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:00.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.866+0000 7f9cd1c5e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:00.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.866+0000 7f9cd1c5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:00.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.866+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ccc13f770 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.940 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.940+0000 7f9cb3fff640 1 -- 172.21.15.87:0/582085016 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+8539 (secure 0 0 0) 0x7f9cc005dcb0 con 0x7f9ccc0b92d0 2024-07-27T14:41:00.941 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.940+0000 7f9cb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:00.941 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.940+0000 7f9cb3fff640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:00.941 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:41:00.941 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":13,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:40:59.264180+0000","last_up_change":"2024-07-27T14:40:59.264180+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.942+0000 7f9cd1c5e640 10 librados: watch_flush enter 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 10 librados: watch_flush exit 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 20 client.4233.objecter shutdown clearing up homeless session... 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 10 client.4233.objecter successfully canceled tick 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 msgr2=0x7f9cb806fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 0x7f9cb806fc20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc036000 tx=0x7f9cbc03d040 comp rx=0 tx=0).stop 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 10 monclient: shutdown 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 msgr2=0x7f9ccc13cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000e930 tx=0x7f9cc000edf0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 shutdown_connections 2024-07-27T14:41:00.944 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9cb806d760 0x7f9cb806fc20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ccc140050 0x7f9ccc13cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ccc0b92d0 0x7f9ccc13cab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 --2- 172.21.15.87:0/582085016 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ccc0b2d80 0x7f9ccc14a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 >> 172.21.15.87:0/582085016 conn(0x7f9ccc067d80 msgr2=0x7f9ccc0b5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 shutdown_connections 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 -- 172.21.15.87:0/582085016 wait complete. 2024-07-27T14:41:00.945 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:00.943+0000 7f9cd1c5e640 1 librados: shutdown 2024-07-27T14:41:00.954 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-07-27T14:41:06.554 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.531+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-07-27T14:41:06.678 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.556+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-07-27T14:41:06.695 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.678+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-07-27T14:41:06.745 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.729+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 3 2024-07-27T14:41:06.756 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.745+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-07-27T14:41:06.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:06.797+0000 5dce500 -1 bdev(0x164e0810 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-07-27T14:41:06.900 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.756+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-07-27T14:41:07.030 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.799+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-07-27T14:41:07.046 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:06.901+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-07-27T14:41:07.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:06.820+0000 5dce500 -1 bdev(0x164e0810 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-27T14:41:07.304 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:07.046+0000 5dce500 -1 bdev(0x164b1090 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-07-27T14:41:07.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:07.237+0000 5dce500 -1 bdev(0x164e0810 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-27T14:41:07.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:07.248+0000 5dce500 -1 bdev(0x164e0810 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-27T14:41:07.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:07.263+0000 5dce500 -1 bdev(0x164e0810 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-27T14:41:07.761 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:41:07.854 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.854+0000 7f04bbcaa640 10 monclient: get_monmap_and_config 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.854+0000 7f04bbcaa640 10 monclient: build_initial_monmap 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.854+0000 7f04bbcaa640 10 monclient: monmap: 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:41:07.855182+0000 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:41:07.855182+0000 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:07.855 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:07.856 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.856 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.854+0000 7f04bbcaa640 1 Processor -- start 2024-07-27T14:41:07.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 -- start start 2024-07-27T14:41:07.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient: init 2024-07-27T14:41:07.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:07.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-07-27T14:41:07.858 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04b40b07b0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04b413eeb0 0x7f04b40abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-c con 0x7f04b413eeb0 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413c080 0x7f04b413c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-b con 0x7f04b413c080 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04b413f2e0 con 0x7f04b40b07b0 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f04b40b0b90 con 0x7f04b413eeb0 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f04b413fe40 con 0x7f04b413c080 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.855+0000 7f04bbcaa640 10 monclient(hunting): authenticate will time out at 1207.881817s 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9a46640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f04b40b07b0 0x7f04b40ac6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:41986/0 (socket says 172.21.15.87:41986) 2024-07-27T14:41:07.859 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9a46640 1 -- 172.21.15.87:0/423109292 learned_addr learned my addr 172.21.15.87:0/423109292 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9245640 1 --2- 172.21.15.87:0/423109292 >> v2:172.21.15.112:3300/0 conn(0x7f04b413eeb0 0x7f04b40abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04ba247640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413c080 0x7f04b413c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9245640 10 monclient(hunting): get_auth_request con 0x7f04b413eeb0 auth_method 0 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9245640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04b9245640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04ba247640 10 monclient(hunting): get_auth_request con 0x7f04b413c080 auth_method 0 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04ba247640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04ba247640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.856+0000 7f04ba247640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:07.860 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:07.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:07.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:07.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.857+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:07.861 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 1 -- 172.21.15.87:0/423109292 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413c080 msgr2=0x7f04b413c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413c080 0x7f04b413c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 1 -- 172.21.15.87:0/423109292 >> v1:172.21.15.87:6789/0 conn(0x7f04b40b07b0 legacy=0x7f04b40ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 10 monclient: _finish_hunting 0 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 1 monclient: found mon.noname-c 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.858+0000 7f04b9245640 1 -- 172.21.15.87:0/423109292 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4073da0 con 0x7f04b413eeb0 2024-07-27T14:41:07.862 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.860+0000 7f04b9245640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b413eeb0 0x7f04b40abb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04a40097a0 tx=0x7f04a402da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4b94597ac672e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 1 -- 172.21.15.87:0/423109292 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031870 con 0x7f04b413eeb0 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 10 monclient: dump: 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:07.863 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 10 monclient: _finish_auth 0 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 10 monclient: _check_auth_tickets 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04b8a44640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04bbcaa640 5 monclient: authenticate success, global_id 4267 2024-07-27T14:41:07.864 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.862+0000 7f04bbcaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 1 -- 172.21.15.87:0/423109292 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4031a10 con 0x7f04b413eeb0 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 1 -- 172.21.15.87:0/423109292 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031de0 con 0x7f04b413eeb0 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 4 set_mon_vals no callback set 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04b8a44640 10 monclient: dump: 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:07.866 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 10 monclient: shutdown 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 -- 172.21.15.87:0/423109292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b413eeb0 msgr2=0x7f04b40abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b413eeb0 0x7f04b40abb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04a40097a0 tx=0x7f04a402da50 comp rx=0 tx=0).stop 2024-07-27T14:41:07.867 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 -- 172.21.15.87:0/423109292 shutdown_connections 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413c080 0x7f04b413c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/423109292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b413eeb0 0x7f04b40abb80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 -- 172.21.15.87:0/423109292 >> 172.21.15.87:0/423109292 conn(0x7f04b40b0da0 msgr2=0x7f04b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 -- 172.21.15.87:0/423109292 shutdown_connections 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.864+0000 7f04bbcaa640 1 -- 172.21.15.87:0/423109292 wait complete. 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.865+0000 7f04bbcaa640 10 monclient: build_initial_monmap 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.865+0000 7f04bbcaa640 10 monclient: monmap: 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.868 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.865+0000 7f04bbcaa640 1 librados: starting msgr at 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.865+0000 7f04bbcaa640 1 librados: starting objecter 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 Processor -- start 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 -- start start 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 librados: setting wanted keys 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 librados: calling monclient init 2024-07-27T14:41:07.869 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient: init 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-b con 0x7f04b40b07b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b413c080 0x7f04b413b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.872 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-a con 0x7f04b413c080 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413eeb0 0x7f04b413b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): picked mon.noname-c con 0x7f04b413eeb0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f04b40ad0a0 con 0x7f04b413c080 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f04b40acf20 con 0x7f04b40b07b0 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f04b40afcb0 con 0x7f04b413eeb0 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.866+0000 7f04bbcaa640 10 monclient(hunting): authenticate will time out at 1207.892737s 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9245640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b413c080 0x7f04b413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.873 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04ba247640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413eeb0 0x7f04b413b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:38134/0 (socket says 172.21.15.87:38134) 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 1 -- 172.21.15.87:0/2372807286 learned_addr learned my addr 172.21.15.87:0/2372807286 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): get_auth_request con 0x7f04b40b07b0 auth_method 0 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04ba247640 10 monclient(hunting): get_auth_request con 0x7f04b413eeb0 auth_method 0 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04ba247640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04ba247640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04ba247640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9245640 10 monclient(hunting): get_auth_request con 0x7f04b413c080 auth_method 0 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:07.874 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9245640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9245640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.867+0000 7f04b9a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04ba247640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.868+0000 7f04b9245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.870+0000 7f04b9a46640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.870+0000 7f04b9a46640 1 -- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413eeb0 msgr2=0x7f04b413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.870+0000 7f04b9a46640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413eeb0 0x7f04b413b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:07.875 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.870+0000 7f04b9a46640 1 -- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b413c080 msgr2=0x7f04b413b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.870+0000 7f04b9a46640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b413c080 0x7f04b413b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.871+0000 7f04b9a46640 10 monclient: _finish_hunting 0 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.871+0000 7f04b9a46640 1 monclient: found mon.noname-b 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.871+0000 7f04b9a46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.871+0000 7f04b9a46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.871+0000 7f04b9a46640 1 -- 172.21.15.87:0/2372807286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4142ca0 con 0x7f04b40b07b0 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.873+0000 7f04b9a46640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04b000c400 tx=0x7f04b000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.873+0000 7f04aa7fc640 10 client.?.objecter ms_handle_connect 0x7f04b40b07b0 2024-07-27T14:41:07.876 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.873+0000 7f04aa7fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0017070 con 0x7f04b40b07b0 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: dump: 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:07.878 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b0005ce0 con 0x7f04b40b07b0 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 5 monclient: authenticate success, global_id 4270 2024-07-27T14:41:07.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 10 monclient: _renew_subs 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b4142f70 con 0x7f04b40b07b0 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00072f0 con 0x7f04b40b07b0 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04aa7fc640 10 monclient: dump: 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:07.880 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 10 monclient: _renew_subs 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b4141060 con 0x7f04b40b07b0 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04bbcaa640 1 librados: init done 2024-07-27T14:41:07.881 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.877+0000 7f04abfff640 4 set_mon_vals no callback set 2024-07-27T14:41:07.882 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.878+0000 7f04bbcaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:07.882 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.878+0000 7f04bbcaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:07.882 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.878+0000 7f04bbcaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:07.882 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.878+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b40b00c0 con 0x7f04b40b07b0 2024-07-27T14:41:07.897 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.896+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f04b001a070 con 0x7f04b40b07b0 2024-07-27T14:41:07.897 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.897+0000 7f04aa7fc640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 0x7f049806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:07.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.897+0000 7f04b9245640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 0x7f049806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:07.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.897+0000 7f04b9245640 10 monclient: get_auth_request con 0x7f049806d780 auth_method 0 2024-07-27T14:41:07.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.898+0000 7f04aa7fc640 10 client.4270.objecter ms_handle_connect 0x7f049806d780 2024-07-27T14:41:07.898 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:07.898+0000 7f04b9245640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 0x7f049806fc40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04a4038000 tx=0x7f04a403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:08.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.137+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4297+0+0 (secure 0 0 0) 0x7f04b0056a20 con 0x7f04b40b07b0 2024-07-27T14:41:08.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.137+0000 7f04aa7fc640 10 client.4270.objecter ms_dispatch 0x7f04b4000b90 osd_map(17..17 src has 1..17) 2024-07-27T14:41:08.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.137+0000 7f04aa7fc640 3 client.4270.objecter handle_osd_map got epochs [17,17] > 0 2024-07-27T14:41:08.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.137+0000 7f04aa7fc640 3 client.4270.objecter handle_osd_map decoding full epoch 17 2024-07-27T14:41:08.138 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.137+0000 7f04aa7fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-07-27T14:41:08.148 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.148+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f04b0021020 con 0x7f04b40b07b0 2024-07-27T14:41:08.148 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.148+0000 7f04aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:08.148 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.148+0000 7f04aa7fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:08.324 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:08.321+0000 ac84640 -1 osd.6 0 waiting for initial osdmap 2024-07-27T14:41:08.430 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.429+0000 7f04bbcaa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:08.430 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.429+0000 7f04bbcaa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:08.430 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.429+0000 7f04bbcaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:08.430 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.429+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04b41d7d50 con 0x7f04b40b07b0 2024-07-27T14:41:08.516 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.515+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f04b0054ff0 con 0x7f04b40b07b0 2024-07-27T14:41:08.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.681+0000 7f04aa7fc640 1 -- 172.21.15.87:0/2372807286 <== mon.1 v2:172.21.15.112:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+11891 (secure 0 0 0) 0x7f04b4142f70 con 0x7f04b40b07b0 2024-07-27T14:41:08.681 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.681+0000 7f04aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:08.682 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.681+0000 7f04aa7fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:08.682 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:41:08.682 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":17,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:41:04.215091+0000","last_up_change":"2024-07-27T14:41:01.948269+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T14:41:00.102594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6809","nonce":3385608780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6811","nonce":3385608780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6815","nonce":3385608780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6813","nonce":3385608780}]},"public_addr":"172.21.15.87:6809/3385608780","cluster_addr":"172.21.15.87:6811/3385608780","heartbeat_back_addr":"172.21.15.87:6815/3385608780","heartbeat_front_addr":"172.21.15.87:6813/3385608780","state":["exists","up"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6801","nonce":1509783986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6803","nonce":1509783986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6807","nonce":1509783986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6805","nonce":1509783986}]},"public_addr":"172.21.15.87:6801/1509783986","cluster_addr":"172.21.15.87:6803/1509783986","heartbeat_back_addr":"172.21.15.87:6807/1509783986","heartbeat_front_addr":"172.21.15.87:6805/1509783986","state":["exists","up"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6817","nonce":1100053525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6819","nonce":1100053525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6826","nonce":1100053525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6823","nonce":1100053525}]},"public_addr":"172.21.15.87:6817/1100053525","cluster_addr":"172.21.15.87:6819/1100053525","heartbeat_back_addr":"172.21.15.87:6826/1100053525","heartbeat_front_addr":"172.21.15.87:6823/1100053525","state":["exists","up"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.359413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.864798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.219576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.792415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.790276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.527378+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 10 librados: watch_flush enter 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 10 librados: watch_flush exit 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 20 client.4270.objecter shutdown clearing up homeless session... 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 10 client.4270.objecter successfully canceled tick 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 msgr2=0x7f049806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 0x7f049806fc40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04a4038000 tx=0x7f04a403b040 comp rx=0 tx=0).stop 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 10 monclient: shutdown 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 msgr2=0x7f04b4140b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:08.684 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04b000c400 tx=0x7f04b000c8c0 comp rx=0 tx=0).stop 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 shutdown_connections 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f049806d780 0x7f049806fc40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04b413eeb0 0x7f04b413b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b413c080 0x7f04b413b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 --2- 172.21.15.87:0/2372807286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40b07b0 0x7f04b4140b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 >> 172.21.15.87:0/2372807286 conn(0x7f04b40b0da0 msgr2=0x7f04b4072b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 shutdown_connections 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 -- 172.21.15.87:0/2372807286 wait complete. 2024-07-27T14:41:08.685 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:08.683+0000 7f04bbcaa640 1 librados: shutdown 2024-07-27T14:41:08.694 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-07-27T14:41:08.824 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:41:08.810+0000 14268640 -1 osd.6 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:41:08.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:08.930+0000 ac84640 -1 osd.2 0 waiting for initial osdmap 2024-07-27T14:41:09.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:41:09.389+0000 14268640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T14:41:11.303 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:14.170 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.336 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:14.188 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.361 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:14.228 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.395 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:14.247 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.527 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:14.394 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.599 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:14.466 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.621 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:14.473 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.627 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:14.493 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:11.859 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:14.725 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:15.500 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-27T14:41:15.601 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.601+0000 7ff084b5e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.601+0000 7ff084b5e640 10 monclient: build_initial_monmap 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.601+0000 7ff084b5e640 10 monclient: monmap: 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:41:15.602454+0000 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:41:15.602454+0000 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:15.602 INFO:teuthology.misc.health.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:15.603 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.601+0000 7ff084b5e640 1 Processor -- start 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 1 -- start start 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient: init 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-07-27T14:41:15.604 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0800afcb0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0800b0090 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ff080072da0 0x7ff080073210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-a con 0x7ff080072da0 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0800adec0 con 0x7ff0800afcb0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0800b0b10 con 0x7ff0800b0090 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7ff0800ada00 con 0x7ff080072da0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.602+0000 7ff084b5e640 10 monclient(hunting): authenticate will time out at 1215.628869s 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.603+0000 7ff07ed9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ff080072da0 0x7ff080073210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.603+0000 7ff07dd9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff0800b0090 0x7ff0800ae3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:59900/0 (socket says 172.21.15.87:59900) 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.603+0000 7ff07dd9b640 1 -- 172.21.15.87:0/3276551283 learned_addr learned my addr 172.21.15.87:0/3276551283 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:15.605 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.604+0000 7ff07ed9d640 10 monclient(hunting): get_auth_request con 0x7ff080072da0 auth_method 0 2024-07-27T14:41:15.606 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.604+0000 7ff07ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:15.606 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.604+0000 7ff07ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:15.606 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.604+0000 7ff07ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:15.606 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.605+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:15.606 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.605+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.605+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 1 -- 172.21.15.87:0/3276551283 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2218537142 0 0) 0x7ff0800b0b10 con 0x7ff0800b0090 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 10 monclient(hunting): _negotiate 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 10 monclient(hunting): my global_id is 4276 2024-07-27T14:41:15.607 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.606+0000 7ff07d59a640 1 -- 172.21.15.87:0/3276551283 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff068003520 con 0x7ff0800b0090 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 1 -- 172.21.15.87:0/3276551283 >> v1:172.21.15.112:6789/0 conn(0x7ff0800b0090 legacy=0x7ff0800ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 1 -- 172.21.15.87:0/3276551283 >> v1:172.21.15.87:6789/0 conn(0x7ff0800afcb0 legacy=0x7ff080143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 10 monclient: _finish_hunting 0 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 1 monclient: found mon.noname-a 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:15.608 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:15.609 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.608+0000 7ff07ed9d640 1 -- 172.21.15.87:0/3276551283 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0800adb80 con 0x7ff080072da0 2024-07-27T14:41:15.610 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.610+0000 7ff07ed9d640 1 --2- 172.21.15.87:0/3276551283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff080073210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff070009d90 tx=0x7ff07002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1dcee1057b1b3d8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 1 -- 172.21.15.87:0/3276551283 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070039070 con 0x7ff080072da0 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: dump: 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.615 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: _finish_auth 0 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: _check_auth_tickets 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 1 -- 172.21.15.87:0/3276551283 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff07002cd00 con 0x7ff080072da0 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 1 -- 172.21.15.87:0/3276551283 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070034660 con 0x7ff080072da0 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff084b5e640 5 monclient: authenticate success, global_id 4251 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff084b5e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff084b5e640 4 set_mon_vals no callback set 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:15.616 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.614+0000 7ff07d59a640 10 monclient: dump: 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:15.617 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.620 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 10 monclient: shutdown 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 -- 172.21.15.87:0/3276551283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 msgr2=0x7ff080073210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 --2- 172.21.15.87:0/3276551283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff080073210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff070009d90 tx=0x7ff07002bf50 comp rx=0 tx=0).stop 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 -- 172.21.15.87:0/3276551283 shutdown_connections 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 --2- 172.21.15.87:0/3276551283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff080073210 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 -- 172.21.15.87:0/3276551283 >> 172.21.15.87:0/3276551283 conn(0x7ff0800b0da0 msgr2=0x7ff0800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 -- 172.21.15.87:0/3276551283 shutdown_connections 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.615+0000 7ff084b5e640 1 -- 172.21.15.87:0/3276551283 wait complete. 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.616+0000 7ff084b5e640 10 monclient: build_initial_monmap 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.616+0000 7ff084b5e640 10 monclient: monmap: 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-07-27T14:41:15.621 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 librados: starting msgr at 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 librados: starting objecter 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 Processor -- start 2024-07-27T14:41:15.622 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 -- start start 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 librados: setting wanted keys 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 librados: calling monclient init 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient: init 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-a con 0x7ff080072da0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0800afcb0 0x7ff080070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0800afcb0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff08013e880 0x7ff080142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): picked mon.noname-c con 0x7ff08013e880 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:15.623 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff08014c340 con 0x7ff080072da0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff08014c1c0 con 0x7ff0800afcb0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff08014c4c0 con 0x7ff08013e880 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.617+0000 7ff084b5e640 10 monclient(hunting): authenticate will time out at 1215.643723s 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0800afcb0 0x7ff080070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07ed9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff08013e880 0x7ff080142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55916/0 (socket says 172.21.15.87:55916) 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 1 -- 172.21.15.87:0/2773904848 learned_addr learned my addr 172.21.15.87:0/2773904848 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 10 monclient(hunting): get_auth_request con 0x7ff080072da0 auth_method 0 2024-07-27T14:41:15.624 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07dd9b640 10 monclient(hunting): get_auth_request con 0x7ff0800afcb0 auth_method 0 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07ed9d640 10 monclient(hunting): get_auth_request con 0x7ff08013e880 auth_method 0 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.618+0000 7ff07ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:15.625 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:15.626 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.619+0000 7ff07ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 -- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff08013e880 msgr2=0x7ff080142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff08013e880 0x7ff080142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 -- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0800afcb0 msgr2=0x7ff080070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0800afcb0 0x7ff080070730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 10 monclient: _finish_hunting 0 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 monclient: found mon.noname-a 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:15.627 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:15.628 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.621+0000 7ff07e59c640 1 -- 172.21.15.87:0/2773904848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff080071d60 con 0x7ff080072da0 2024-07-27T14:41:15.628 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.624+0000 7ff07e59c640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff07800c400 tx=0x7ff07800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:15.628 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.624+0000 7ff06effd640 10 client.?.objecter ms_handle_connect 0x7ff080072da0 2024-07-27T14:41:15.628 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.624+0000 7ff06effd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:15.629 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0780052d0 con 0x7ff080072da0 2024-07-27T14:41:15.629 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:15.629 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:15.629 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: dump: 2024-07-27T14:41:15.629 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: _finish_auth 0 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:15.630 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 5 monclient: authenticate success, global_id 4257 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff078005490 con 0x7ff080072da0 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff078007260 con 0x7ff080072da0 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff06effd640 10 monclient: dump: 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:15.631 INFO:teuthology.misc.health.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr: 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 10 monclient: _renew_subs 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff07cd99640 4 set_mon_vals no callback set 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff080071fd0 con 0x7ff080072da0 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 10 monclient: _renew_subs 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08006fc50 con 0x7ff080072da0 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.627+0000 7ff084b5e640 1 librados: init done 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.628+0000 7ff084b5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.628+0000 7ff084b5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:15.632 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.628+0000 7ff084b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:15.633 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.628+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff080073210 con 0x7ff080072da0 2024-07-27T14:41:15.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.877+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff078007400 con 0x7ff080072da0 2024-07-27T14:41:15.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.877+0000 7ff06effd640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 0x7ff06006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:15.878 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.878+0000 7ff07dd9b640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 0x7ff06006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:15.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.878+0000 7ff07dd9b640 10 monclient: get_auth_request con 0x7ff06006d810 auth_method 0 2024-07-27T14:41:15.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.878+0000 7ff07dd9b640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 0x7ff06006fcd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff0801361d0 tx=0x7ff074007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:15.879 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.878+0000 7ff06effd640 10 client.4257.objecter ms_handle_connect 0x7ff06006d810 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7ff07805f330 con 0x7ff080072da0 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 10 client.4257.objecter ms_dispatch 0x7ff080000b90 osd_map(19..19 src has 1..19) 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 3 client.4257.objecter handle_osd_map got epochs [19,19] > 0 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 3 client.4257.objecter handle_osd_map decoding full epoch 19 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 20 client.4257.objecter dump_active .. 0 homeless 2024-07-27T14:41:15.890 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.889+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7ff078058b40 con 0x7ff080072da0 2024-07-27T14:41:15.891 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.890+0000 7ff06effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:15.891 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:15.890+0000 7ff06effd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:16.076 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.076+0000 7ff084b5e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:16.076 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.076+0000 7ff084b5e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:16.076 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.076+0000 7ff084b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:16.076 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.076+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0800758a0 con 0x7ff080072da0 2024-07-27T14:41:16.163 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.163+0000 7ff06effd640 1 -- 172.21.15.87:0/2773904848 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12873 (secure 0 0 0) 0x7ff07805ce40 con 0x7ff080072da0 2024-07-27T14:41:16.163 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.163+0000 7ff06effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:16.163 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.163+0000 7ff06effd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:16.164 INFO:teuthology.misc.health.smithi087.stdout: 2024-07-27T14:41:16.164 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":19,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:41:11.894975+0000","last_up_change":"2024-07-27T14:41:10.123887+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T14:41:00.102594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6809","nonce":3385608780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6811","nonce":3385608780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6815","nonce":3385608780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6813","nonce":3385608780}]},"public_addr":"172.21.15.87:6809/3385608780","cluster_addr":"172.21.15.87:6811/3385608780","heartbeat_back_addr":"172.21.15.87:6815/3385608780","heartbeat_front_addr":"172.21.15.87:6813/3385608780","state":["exists","up"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6801","nonce":1509783986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6803","nonce":1509783986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6807","nonce":1509783986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6805","nonce":1509783986}]},"public_addr":"172.21.15.87:6801/1509783986","cluster_addr":"172.21.15.87:6803/1509783986","heartbeat_back_addr":"172.21.15.87:6807/1509783986","heartbeat_front_addr":"172.21.15.87:6805/1509783986","state":["exists","up"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6822","nonce":357294290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6825","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6827","nonce":357294290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6831","nonce":357294290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6829","nonce":357294290}]},"public_addr":"172.21.15.87:6822/357294290","cluster_addr":"172.21.15.87:6827/357294290","heartbeat_back_addr":"172.21.15.87:6831/357294290","heartbeat_front_addr":"172.21.15.87:6829/357294290","state":["exists","up"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6817","nonce":1100053525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6819","nonce":1100053525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6826","nonce":1100053525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6823","nonce":1100053525}]},"public_addr":"172.21.15.87:6817/1100053525","cluster_addr":"172.21.15.87:6819/1100053525","heartbeat_back_addr":"172.21.15.87:6826/1100053525","heartbeat_front_addr":"172.21.15.87:6823/1100053525","state":["exists","up"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6803","nonce":591715754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6805","nonce":591715754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6809","nonce":591715754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6807","nonce":591715754}]},"public_addr":"172.21.15.112:6803/591715754","cluster_addr":"172.21.15.112:6805/591715754","heartbeat_back_addr":"172.21.15.112:6809/591715754","heartbeat_front_addr":"172.21.15.112:6807/591715754","state":["exists","up"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.359413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.864798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.468303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.219576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.792415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.790276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.126847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.527378+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 10 librados: watch_flush enter 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 10 librados: watch_flush exit 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 20 client.4257.objecter shutdown clearing up homeless session... 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 10 client.4257.objecter successfully canceled tick 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 msgr2=0x7ff06006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 0x7ff06006fcd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff0801361d0 tx=0x7ff074007d30 comp rx=0 tx=0).stop 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 10 monclient: shutdown 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 msgr2=0x7ff0801432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff07800c400 tx=0x7ff07800c8c0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.166 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 shutdown_connections 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff06006d810 0x7ff06006fcd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff08013e880 0x7ff080142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0800afcb0 0x7ff080070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 --2- 172.21.15.87:0/2773904848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff080072da0 0x7ff0801432a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 >> 172.21.15.87:0/2773904848 conn(0x7ff0800b0da0 msgr2=0x7ff0800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 shutdown_connections 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 -- 172.21.15.87:0/2773904848 wait complete. 2024-07-27T14:41:16.167 INFO:teuthology.misc.health.smithi087.stderr:2024-07-27T14:41:16.165+0000 7ff084b5e640 1 librados: shutdown 2024-07-27T14:41:16.176 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-07-27T14:41:16.176 INFO:tasks.ceph:Creating RBD pool 2024-07-27T14:41:16.176 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool create rbd 8 2024-07-27T14:41:16.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.274+0000 7fee4ea0b640 10 monclient: get_monmap_and_config 2024-07-27T14:41:16.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.274+0000 7fee4ea0b640 10 monclient: build_initial_monmap 2024-07-27T14:41:16.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.274+0000 7fee4ea0b640 10 monclient: monmap: 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:16.275259+0000 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:16.275259+0000 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:16.275 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.274+0000 7fee4ea0b640 1 Processor -- start 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 1 -- start start 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient: init 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 0x7fee48146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-b con 0x7fee480678a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:16.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee480b0420 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee480b0e60 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee480736d0 con 0x7fee480b0e60 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee480b0940 con 0x7fee480b0420 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fee48067c80 con 0x7fee480678a0 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.275+0000 7fee4ea0b640 10 monclient(hunting): authenticate will time out at 1216.301648s 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee4cfa8640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fee480b0e60 0x7fee4814d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:50666/0 (socket says 172.21.15.87:50666) 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee4cfa8640 1 -- 172.21.15.87:0/1156192493 learned_addr learned my addr 172.21.15.87:0/1156192493 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee47fff640 1 --2- 172.21.15.87:0/1156192493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 0x7fee48146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:16.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee47fff640 10 monclient(hunting): get_auth_request con 0x7fee480678a0 auth_method 0 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee47fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.276+0000 7fee47fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.277+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.277+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.277+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987525603 0 0) 0x7fee480b0940 con 0x7fee480b0420 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 10 monclient(hunting): _negotiate 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 10 monclient(hunting): my global_id is 4282 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.278+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee30003520 con 0x7fee480b0420 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525153769 0 0) 0x7fee480736d0 con 0x7fee480b0e60 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 10 monclient(hunting): _negotiate 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 10 monclient(hunting): my global_id is 4260 2024-07-27T14:41:16.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee480b0940 con 0x7fee480b0e60 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 1 -- 172.21.15.87:0/1156192493 >> v1:172.21.15.112:6789/0 conn(0x7fee480b0420 legacy=0x7fee4813fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 1 -- 172.21.15.87:0/1156192493 >> v1:172.21.15.87:6789/0 conn(0x7fee480b0e60 legacy=0x7fee4814d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 1 monclient: found mon.noname-b 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.279+0000 7fee47fff640 1 -- 172.21.15.87:0/1156192493 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee48068370 con 0x7fee480678a0 2024-07-27T14:41:16.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.281+0000 7fee47fff640 1 --2- 172.21.15.87:0/1156192493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 0x7fee48146080 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fee380097d0 tx=0x7fee3802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=239836782ddfce7f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38035020 con 0x7fee480678a0 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: dump: 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:16.285 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee38031890 con 0x7fee480678a0 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee4ea0b640 5 monclient: authenticate success, global_id 4244 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee4ea0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee4ea0b640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:16.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 1 -- 172.21.15.87:0/1156192493 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38031c60 con 0x7fee480678a0 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee4ea0b640 4 set_mon_vals no callback set 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.284+0000 7fee46ffd640 10 monclient: dump: 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:16.287 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:16.288 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 10 monclient: shutdown 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1156192493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 msgr2=0x7fee48146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1156192493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 0x7fee48146080 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fee380097d0 tx=0x7fee3802b930 comp rx=0 tx=0).stop 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1156192493 shutdown_connections 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1156192493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480678a0 0x7fee48146080 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1156192493 >> 172.21.15.87:0/1156192493 conn(0x7fee480b6260 msgr2=0x7fee4806b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1156192493 shutdown_connections 2024-07-27T14:41:16.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.285+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1156192493 wait complete. 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.286+0000 7fee4ea0b640 10 monclient: build_initial_monmap 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.286+0000 7fee4ea0b640 10 monclient: monmap: 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.286+0000 7fee4ea0b640 1 librados: starting msgr at 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.286+0000 7fee4ea0b640 1 librados: starting objecter 2024-07-27T14:41:16.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.286+0000 7fee4ea0b640 1 Processor -- start 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 -- start start 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 librados: setting wanted keys 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 librados: calling monclient init 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient: init 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 0x7fee480706e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-a con 0x7fee480678a0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480b0420 0x7fee48144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-c con 0x7fee480b0420 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): picked mon.noname-b con 0x7fee480b0e60 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:16.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee48140750 con 0x7fee480678a0 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee481465c0 con 0x7fee480b0e60 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fee48072510 con 0x7fee480b0420 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.287+0000 7fee4ea0b640 10 monclient(hunting): authenticate will time out at 1216.313862s 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee47fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 0x7fee480706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:42604/0 (socket says 172.21.15.87:42604) 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 1 -- 172.21.15.87:0/1843966618 learned_addr learned my addr 172.21.15.87:0/1843966618 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 10 monclient(hunting): get_auth_request con 0x7fee480b0e60 auth_method 0 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:16.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee4cfa8640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee47fff640 10 monclient(hunting): get_auth_request con 0x7fee480678a0 auth_method 0 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee47fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.288+0000 7fee47fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee4cfa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee4cfa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee4cfa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.289+0000 7fee47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:16.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.290+0000 7fee477fe640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480b0420 0x7fee48144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:16.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.291+0000 7fee477fe640 10 monclient(hunting): get_auth_request con 0x7fee480b0420 auth_method 0 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.291+0000 7fee477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.291+0000 7fee477fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.291+0000 7fee477fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 -- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480b0420 msgr2=0x7fee48144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480b0420 0x7fee48144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 -- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 msgr2=0x7fee480706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 0x7fee480706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 10 monclient: _finish_hunting 0 2024-07-27T14:41:16.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 monclient: found mon.noname-b 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee4cfa8640 1 -- 172.21.15.87:0/1843966618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee480727f0 con 0x7fee480b0e60 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.292+0000 7fee47fff640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 0x7fee480706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.295+0000 7fee4cfa8640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee4000caa0 tx=0x7fee4000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.295+0000 7fee44ff9640 10 client.?.objecter ms_handle_connect 0x7fee480b0e60 2024-07-27T14:41:16.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.295+0000 7fee44ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:16.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee40017070 con 0x7fee480b0e60 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: dump: 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.303 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee4000ac00 con 0x7fee480b0e60 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 5 monclient: authenticate success, global_id 4285 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee40012400 con 0x7fee480b0e60 2024-07-27T14:41:16.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee44ff9640 10 monclient: dump: 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:16.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 10 monclient: _renew_subs 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee48071350 con 0x7fee480b0e60 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee467fc640 4 set_mon_vals no callback set 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 10 monclient: _renew_subs 2024-07-27T14:41:16.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:16.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee48070c20 con 0x7fee480b0e60 2024-07-27T14:41:16.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.302+0000 7fee4ea0b640 1 librados: init done 2024-07-27T14:41:16.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.303+0000 7fee4ea0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:16.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.303+0000 7fee4ea0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:16.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.303+0000 7fee4ea0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:16.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.303+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee480719d0 con 0x7fee480b0e60 2024-07-27T14:41:16.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.546+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee4001a070 con 0x7fee480b0e60 2024-07-27T14:41:16.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.546+0000 7fee44ff9640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 0x7fee3406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:16.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.546+0000 7fee47fff640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 0x7fee3406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:16.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.546+0000 7fee47fff640 10 monclient: get_auth_request con 0x7fee3406d810 auth_method 0 2024-07-27T14:41:16.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.547+0000 7fee47fff640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 0x7fee3406fcd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:16.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.547+0000 7fee44ff9640 10 client.4285.objecter ms_handle_connect 0x7fee3406d810 2024-07-27T14:41:16.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.548+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7fee4005fb30 con 0x7fee480b0e60 2024-07-27T14:41:16.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.548+0000 7fee44ff9640 10 client.4285.objecter ms_dispatch 0x7fee48000b90 osd_map(19..19 src has 1..19) 2024-07-27T14:41:16.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.548+0000 7fee44ff9640 3 client.4285.objecter handle_osd_map got epochs [19,19] > 0 2024-07-27T14:41:16.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.548+0000 7fee44ff9640 3 client.4285.objecter handle_osd_map decoding full epoch 19 2024-07-27T14:41:16.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.548+0000 7fee44ff9640 20 client.4285.objecter dump_active .. 0 homeless 2024-07-27T14:41:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.559+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fee40021020 con 0x7fee480b0e60 2024-07-27T14:41:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.559+0000 7fee44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.559+0000 7fee44ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:16.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.735+0000 7fee4ea0b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-27T14:41:16.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.735+0000 7fee4ea0b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-27T14:41:16.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.735+0000 7fee4ea0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:16.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:16.735+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fee480afdf0 con 0x7fee480b0e60 2024-07-27T14:41:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.093+0000 7fee44ff9640 1 -- 172.21.15.87:0/1843966618 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v20) ==== 109+0+0 (secure 0 0 0) 0x7fee48071350 con 0x7fee480b0e60 2024-07-27T14:41:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.093+0000 7fee44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-27T14:41:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.093+0000 7fee44ff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-07-27T14:41:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 10 librados: watch_flush enter 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 10 librados: watch_flush exit 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 20 client.4285.objecter shutdown clearing up homeless session... 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 10 client.4285.objecter successfully canceled tick 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 msgr2=0x7fee3406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 0x7fee3406fcd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38008030 comp rx=0 tx=0).stop 2024-07-27T14:41:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 10 monclient: shutdown 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 msgr2=0x7fee48144640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee4000caa0 tx=0x7fee4000cf60 comp rx=0 tx=0).stop 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 shutdown_connections 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee3406d810 0x7fee3406fcd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee480b0e60 0x7fee48144640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee480b0420 0x7fee48144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 --2- 172.21.15.87:0/1843966618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee480678a0 0x7fee480706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 >> 172.21.15.87:0/1843966618 conn(0x7fee480b6260 msgr2=0x7fee4813fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 shutdown_connections 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 -- 172.21.15.87:0/1843966618 wait complete. 2024-07-27T14:41:17.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.095+0000 7fee4ea0b640 1 librados: shutdown 2024-07-27T14:41:17.107 DEBUG:teuthology.orchestra.run.smithi087:> rbd --cluster ceph pool init rbd 2024-07-27T14:41:17.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.156+0000 7f3e7600ed80 10 monclient: get_monmap_and_config 2024-07-27T14:41:17.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.156+0000 7f3e7600ed80 10 monclient: build_initial_monmap 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.156+0000 7f3e7600ed80 10 monclient: monmap: 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:17.157249+0000 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:17.157249+0000 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:17.157 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.156+0000 7f3e7600ed80 1 Processor -- start 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 -- start start 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient: init 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient: _reopen_session rank -1 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 --2- >> v2:172.21.15.87:3300/0 conn(0x558a1e72abe0 0x558a1e7b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-a con 0x558a1e72abe0 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-b con 0x558a1e725960 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 --2- >> v2:172.21.15.112:3300/0 conn(0x558a1e7b8c90 0x558a1e7b9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-c con 0x558a1e7b8c90 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:17.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x558a1e581500 con 0x558a1e72abe0 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x558a1e5e5e40 con 0x558a1e7b8c90 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x558a1e5e1e00 con 0x558a1e725960 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): _renew_subs 2024-07-27T14:41:17.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.157+0000 7f3e7600ed80 10 monclient(hunting): authenticate will time out at 1217.183597s 2024-07-27T14:41:17.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.169+0000 7f3e73674640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:17.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60432/0 (socket says 172.21.15.87:60432) 2024-07-27T14:41:17.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 1 -- 172.21.15.87:0/2771285208 learned_addr learned my addr 172.21.15.87:0/2771285208 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): get_auth_request con 0x558a1e725960 auth_method 0 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.170+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:17.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 -- 172.21.15.87:0/2771285208 >> v2:172.21.15.112:3300/0 conn(0x558a1e7b8c90 msgr2=0x558a1e7b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 --2- 172.21.15.87:0/2771285208 >> v2:172.21.15.112:3300/0 conn(0x558a1e7b8c90 0x558a1e7b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 -- 172.21.15.87:0/2771285208 >> v2:172.21.15.87:3300/0 conn(0x558a1e72abe0 msgr2=0x558a1e7b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 --2- 172.21.15.87:0/2771285208 >> v2:172.21.15.87:3300/0 conn(0x558a1e72abe0 0x558a1e7b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 10 monclient: _finish_hunting 0 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 monclient: found mon.noname-b 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:17.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.172+0000 7f3e73674640 1 -- 172.21.15.87:0/2771285208 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1e566ce0 con 0x558a1e725960 2024-07-27T14:41:17.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.176+0000 7f3e73674640 1 --2- 172.21.15.87:0/2771285208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x558a1e72b310 tx=0x7f3e5c01cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778ec7679f515d4c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 1 -- 172.21.15.87:0/2771285208 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c057070 con 0x558a1e725960 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 10 monclient: dump: 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:17.179 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 10 monclient: _finish_auth 0 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 10 monclient: _check_auth_tickets 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e72e73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e7600ed80 5 monclient: authenticate success, global_id 4250 2024-07-27T14:41:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.179+0000 7f3e7600ed80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 1 -- 172.21.15.87:0/2771285208 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5c052640 con 0x558a1e725960 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e7600ed80 10 monclient: get_monmap_and_config success 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 1 -- 172.21.15.87:0/2771285208 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c052a10 con 0x558a1e725960 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e7600ed80 4 set_mon_vals no callback set 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.180+0000 7f3e72e73640 10 monclient: dump: 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.183 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 10 monclient: shutdown 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 -- 172.21.15.87:0/2771285208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 msgr2=0x558a1e7b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/2771285208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x558a1e72b310 tx=0x7f3e5c01cf70 comp rx=0 tx=0).stop 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 -- 172.21.15.87:0/2771285208 shutdown_connections 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/2771285208 >> v2:172.21.15.112:3300/0 conn(0x558a1e7b8c90 0x558a1e7b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/2771285208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e725960 0x558a1e7b2c30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/2771285208 >> v2:172.21.15.87:3300/0 conn(0x558a1e72abe0 0x558a1e7b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 -- 172.21.15.87:0/2771285208 >> 172.21.15.87:0/2771285208 conn(0x558a1e6f9a80 msgr2=0x558a1e72a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 -- 172.21.15.87:0/2771285208 shutdown_connections 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.181+0000 7f3e7600ed80 1 -- 172.21.15.87:0/2771285208 wait complete. 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 10 monclient: build_initial_monmap 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 10 monclient: monmap: 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:17.185 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 librados: starting msgr at 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 librados: starting objecter 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 Processor -- start 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 -- start start 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 librados: setting wanted keys 2024-07-27T14:41:17.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 1 librados: calling monclient init 2024-07-27T14:41:17.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.182+0000 7f3e7600ed80 10 monclient: init 2024-07-27T14:41:17.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient: _reopen_session rank -1 2024-07-27T14:41:17.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:41:17.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 0x558a1e7cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-c con 0x558a1e72abe0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:17.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a1e7b8c90 0x558a1e7a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-b con 0x558a1e7b8c90 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a1e7cca80 0x558a1e7a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): picked mon.noname-a con 0x558a1e7cca80 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558a1e725df0 con 0x558a1e7cca80 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a1e7b9890 con 0x558a1e7b8c90 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x558a1e726ae0 con 0x558a1e72abe0 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): _renew_subs 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e7600ed80 10 monclient(hunting): authenticate will time out at 1217.209518s 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.183+0000 7f3e73e75640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 0x558a1e7cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:17.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a1e7b8c90 0x558a1e7a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 0x558a1e7cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60444/0 (socket says 172.21.15.87:60444) 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 1 -- 172.21.15.87:0/1947113049 learned_addr learned my addr 172.21.15.87:0/1947113049 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): get_auth_request con 0x558a1e72abe0 auth_method 0 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): get_auth_request con 0x558a1e7b8c90 auth_method 0 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73674640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.184+0000 7f3e73e75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 -- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a1e7b8c90 msgr2=0x558a1e7a9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a1e7b8c90 0x558a1e7a9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 -- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a1e7cca80 msgr2=0x558a1e7a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a1e7cca80 0x558a1e7a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 10 monclient: _finish_hunting 0 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 monclient: found mon.noname-c 2024-07-27T14:41:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73e75640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1e581c20 con 0x558a1e72abe0 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.187+0000 7f3e73674640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a1e7b8c90 0x558a1e7a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.190+0000 7f3e73e75640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 0x558a1e7cc540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e68008530 tx=0x7f3e680089f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.190+0000 7f3e5bfff640 10 client.?.objecter ms_handle_connect 0x558a1e72abe0 2024-07-27T14:41:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.190+0000 7f3e5bfff640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:17.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e680067a0 con 0x558a1e72abe0 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: dump: 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:17.199 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: _finish_auth 0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e68006940 con 0x558a1e72abe0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 5 monclient: authenticate success, global_id 4253 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 monclient: _renew_subs 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x558a1e71cb40 con 0x558a1e72abe0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68006d30 con 0x558a1e72abe0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e5bfff640 10 monclient: dump: 2024-07-27T14:41:17.200 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e72672640 4 set_mon_vals no callback set 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 monclient: _renew_subs 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x558a1e7be3c0 con 0x558a1e72abe0 2024-07-27T14:41:17.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 1 librados: init done 2024-07-27T14:41:17.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.196+0000 7f3e7600ed80 10 librados: wait_for_osdmap waiting 2024-07-27T14:41:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.218+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3e68019400 con 0x558a1e72abe0 2024-07-27T14:41:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.218+0000 7f3e5bfff640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3e5406b920 0x7f3e5406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.219+0000 7f3e73674640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3e5406b920 0x7f3e5406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.219+0000 7f3e73674640 10 monclient: get_auth_request con 0x7f3e5406b920 auth_method 0 2024-07-27T14:41:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.219+0000 7f3e5bfff640 10 client.4253.objecter ms_handle_connect 0x7f3e5406b920 2024-07-27T14:41:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.219+0000 7f3e73674640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3e5406b920 0x7f3e5406dde0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c054000 tx=0x7f3e5c04e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:17.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5199+0+0 (secure 0 0 0) 0x7f3e680560f0 con 0x558a1e72abe0 2024-07-27T14:41:17.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_map(20..20 src has 1..20) 2024-07-27T14:41:17.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map got epochs [20,20] > 0 2024-07-27T14:41:17.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map decoding full epoch 20 2024-07-27T14:41:17.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e5bfff640 20 client.4253.objecter dump_active .. 0 homeless 2024-07-27T14:41:17.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e7600ed80 10 librados: wait_for_osdmap done waiting 2024-07-27T14:41:17.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e7600ed80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-27T14:41:17.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e7600ed80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-27T14:41:17.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e7600ed80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:17.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:17.221+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x558a1e71af60 con 0x558a1e72abe0 2024-07-27T14:41:18.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.283+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v21) ==== 141+0+0 (secure 0 0 0) 0x7f3e6800d070 con 0x558a1e72abe0 2024-07-27T14:41:18.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.283+0000 7f3e5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-27T14:41:18.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.283+0000 7f3e5bfff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-07-27T14:41:18.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.283+0000 7f3e7600ed80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:18.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.283+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e5c052640 con 0x558a1e72abe0 2024-07-27T14:41:18.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3e68057e20 con 0x558a1e72abe0 2024-07-27T14:41:18.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e5bfff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-07-27T14:41:18.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e72672640 10 client.4253.objecter _get_latest_version latest 21, waiting 2024-07-27T14:41:18.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e72672640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:18.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e72672640 10 monclient: _renew_subs 2024-07-27T14:41:18.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e72672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:18.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.400+0000 7f3e72672640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=21}) -- 0x7f3e4c003c90 con 0x558a1e72abe0 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(21..21 src has 1..21) ==== 658+0+0 (secure 0 0 0) 0x7f3e68056350 con 0x558a1e72abe0 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_map(21..21 src has 1..21) 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map got epochs [21,21] > 20 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map decoding incremental epoch 21 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e5bfff640 20 client.4253.objecter dump_active .. 0 homeless 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 10 librados: create oid=rbd_trash nspace= 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 10 client.4253.objecter _op_submit op 0x558a1e71b6f0 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 20 client.4253.objecter _calc_target epoch 21 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:41:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 20 client.4253.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 10 client.4253.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [3,2] primary 3 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x558a1e7a8830 0x558a1e7a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 20 client.4253.objecter _get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 10 client.4253.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.3 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 20 client.4253.objecter get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 15 client.4253.objecter _session_op_assign 3 1 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 15 client.4253.objecter _send_op 1 to 2.3 on osd.3 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x558a1e7d5b00 con 0x558a1e7a8830 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.403+0000 7f3e7600ed80 5 client.4253.objecter 1 in flight 2024-07-27T14:41:18.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.439+0000 7f3e750d7640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x558a1e7a8830 0x558a1e7a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:18.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.594+0000 7f3e750d7640 10 monclient: get_auth_request con 0x558a1e7a8830 auth_method 0 2024-07-27T14:41:18.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.616+0000 7f3e5bfff640 10 client.4253.objecter ms_handle_connect 0x558a1e7a8830 2024-07-27T14:41:18.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:18.616+0000 7f3e750d7640 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x558a1e7a8830 0x558a1e7a8c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 <== osd.3 v2:172.21.15.87:6816/1100053525 1 ==== osd_op_reply(1 rbd_trash [create] v21'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f3e6400a560 con 0x558a1e7a8830 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_op_reply(1 rbd_trash [create] v21'1 uv1 ondisk = 0) 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 10 client.4253.objecter in handle_osd_op_reply 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 7 client.4253.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 10 client.4253.objecter op 0 rval 0 len 0 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 15 client.4253.objecter handle_osd_op_reply completed tid 1 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 15 client.4253.objecter _finish_op 1 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:19.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 15 client.4253.objecter _session_op_remove 3 1 2024-07-27T14:41:19.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e750d7640 5 client.4253.objecter 0 in flight 2024-07-27T14:41:19.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.416+0000 7f3e7600ed80 10 librados: Objecter returned from create r=0 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 10 client.4253.objecter _op_submit op 0x558a1e827060 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 20 client.4253.objecter _calc_target epoch 21 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 20 client.4253.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 10 client.4253.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 20 client.4253.objecter _get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 10 client.4253.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.3 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 20 client.4253.objecter get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 15 client.4253.objecter _session_op_assign 3 2 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 15 client.4253.objecter _send_op 2 to 2.2 on osd.3 2024-07-27T14:41:19.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x558a1e8275b0 con 0x558a1e7a8830 2024-07-27T14:41:19.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:19.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.418+0000 7f3e7600ed80 5 client.4253.objecter 1 in flight 2024-07-27T14:41:19.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 <== osd.3 v2:172.21.15.87:6816/1100053525 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f3e6400a560 con 0x558a1e7a8830 2024-07-27T14:41:19.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:41:19.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 10 client.4253.objecter in handle_osd_op_reply 2024-07-27T14:41:19.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 7 client.4253.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-07-27T14:41:19.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 10 client.4253.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:41:19.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 10 client.4253.objecter op 0 rval 0 len 0 2024-07-27T14:41:19.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 15 client.4253.objecter handle_osd_op_reply completed tid 2 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 15 client.4253.objecter _finish_op 2 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 15 client.4253.objecter _session_op_remove 3 2 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e750d7640 5 client.4253.objecter 0 in flight 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 10 client.4253.objecter _op_submit op 0x558a1e8261f0 2024-07-27T14:41:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 20 client.4253.objecter _calc_target epoch 21 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 20 client.4253.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 10 client.4253.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 20 client.4253.objecter _get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 10 client.4253.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.3 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 20 client.4253.objecter get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 15 client.4253.objecter _session_op_assign 3 3 2024-07-27T14:41:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 15 client.4253.objecter _send_op 3 to 2.2 on osd.3 2024-07-27T14:41:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x558a1e8266a0 con 0x558a1e7a8830 2024-07-27T14:41:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.440+0000 7f3e7600ed80 5 client.4253.objecter 1 in flight 2024-07-27T14:41:19.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 <== osd.3 v2:172.21.15.87:6816/1100053525 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f3e6400a560 con 0x558a1e7a8830 2024-07-27T14:41:19.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:41:19.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 10 client.4253.objecter in handle_osd_op_reply 2024-07-27T14:41:19.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 7 client.4253.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-07-27T14:41:19.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 10 client.4253.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:41:19.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 10 client.4253.objecter op 0 rval 0 len 0 2024-07-27T14:41:19.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 15 client.4253.objecter handle_osd_op_reply completed tid 3 2024-07-27T14:41:19.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 15 client.4253.objecter _finish_op 3 2024-07-27T14:41:19.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:19.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 15 client.4253.objecter _session_op_remove 3 3 2024-07-27T14:41:19.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e750d7640 5 client.4253.objecter 0 in flight 2024-07-27T14:41:19.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e71e71640 10 client.4253.objecter allocate_selfmanaged_snap; pool: 2 2024-07-27T14:41:19.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e71e71640 10 client.4253.objecter pool_op_submit 4 2024-07-27T14:41:19.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e71e71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:19.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:19.464+0000 7f3e71e71640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f3e500030d0 con 0x558a1e72abe0 2024-07-27T14:41:20.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.183+0000 7f3e5affd640 10 monclient: tick 2024-07-27T14:41:20.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.183+0000 7f3e5affd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:20.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.183+0000 7f3e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:20.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.183+0000 7f3e5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:20.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v23) ==== 55+0+0 (secure 0 0 0) 0x7f3e68086070 con 0x558a1e72abe0 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 pool_op_reply(tid 4 (0) Success v23) 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 client.4253.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v23) 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 client.4253.objecter have request 4 at 0x7f3e50003060 Op: create unmanaged snap 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 20 client.4253.objecter waiting for client to reach epoch 23 before calling back 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 monclient: _renew_subs 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=22}) -- 0x7f3e540720f0 con 0x558a1e72abe0 2024-07-27T14:41:20.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.647+0000 7f3e5bfff640 10 client.4253.objecter done 2024-07-27T14:41:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 10 ==== osd_map(22..23 src has 1..23) ==== 1216+0+0 (secure 0 0 0) 0x7f3e680546c0 con 0x558a1e72abe0 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_map(22..23 src has 1..23) 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map got epochs [22,23] > 21 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map decoding incremental epoch 22 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map decoding incremental epoch 23 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e5bfff640 20 client.4253.objecter dump_active .. 0 homeless 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 librados: queue_aio_write 0x558a1e8225f0 completion 0x558a1e729e30 write_seq 1 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 10 client.4253.objecter _op_submit op 0x7f3e4c006380 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 client.4253.objecter _calc_target epoch 23 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 client.4253.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 10 client.4253.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 client.4253.objecter _get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:20.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 10 client.4253.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.3 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 client.4253.objecter get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 15 client.4253.objecter _session_op_assign 3 5 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 15 client.4253.objecter _send_op 5 to 2.2 on osd.3 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x7f3e4c006850 con 0x558a1e7a8830 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.654+0000 7f3e72672640 5 client.4253.objecter 1 in flight 2024-07-27T14:41:20.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 <== osd.3 v2:172.21.15.87:6816/1100053525 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v23'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f3e6400a560 con 0x558a1e7a8830 2024-07-27T14:41:20.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_op_reply(5 rbd_info [create,write 0~8] v23'1 uv1 ondisk = 0) 2024-07-27T14:41:20.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 10 client.4253.objecter in handle_osd_op_reply 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 7 client.4253.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 10 client.4253.objecter op 0 rval 0 len 0 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 10 client.4253.objecter op 1 rval 0 len 0 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 15 client.4253.objecter handle_osd_op_reply completed tid 5 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 15 client.4253.objecter _finish_op 5 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 15 client.4253.objecter _session_op_remove 3 5 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 5 client.4253.objecter 0 in flight 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e750d7640 20 librados: complete_aio_write 0x558a1e729e30 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e71e71640 10 client.4253.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e71e71640 10 client.4253.objecter pool_op_submit 6 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e71e71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:20.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:20.887+0000 7f3e71e71640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v23) -- 0x7f3e50004520 con 0x558a1e72abe0 2024-07-27T14:41:21.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.183+0000 7f3e5affd640 10 monclient: tick 2024-07-27T14:41:21.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.183+0000 7f3e5affd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:21.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.183+0000 7f3e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v24) ==== 43+0+0 (secure 0 0 0) 0x7f3e68058750 con 0x558a1e72abe0 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 pool_op_reply(tid 6 (0) Success v24) 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 client.4253.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v24) 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 client.4253.objecter have request 6 at 0x7f3e50003060 Op: delete unmanaged snap 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 20 client.4253.objecter waiting for client to reach epoch 24 before calling back 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 monclient: _renew_subs 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=24}) -- 0x7f3e540722c0 con 0x558a1e72abe0 2024-07-27T14:41:21.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.747+0000 7f3e5bfff640 10 client.4253.objecter done 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e5bfff640 1 -- 172.21.15.87:0/1947113049 <== mon.2 v2:172.21.15.87:3301/0 12 ==== osd_map(24..24 src has 1..24) ==== 654+0+0 (secure 0 0 0) 0x7f3e680607a0 con 0x558a1e72abe0 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e5bfff640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_map(24..24 src has 1..24) 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map got epochs [24,24] > 23 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e5bfff640 3 client.4253.objecter handle_osd_map decoding incremental epoch 24 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e5bfff640 20 client.4253.objecter dump_active .. 0 homeless 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 librados: queue_aio_write 0x558a1e8225f0 completion 0x7f3e50001d60 write_seq 2 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 10 client.4253.objecter _op_submit op 0x7f3e4c006380 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 client.4253.objecter _calc_target epoch 24 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:41:21.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 client.4253.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 10 client.4253.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 client.4253.objecter _get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 10 client.4253.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.3 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 client.4253.objecter get_session s=0x558a1e71bba0 osd=3 3 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 15 client.4253.objecter _session_op_assign 3 7 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 15 client.4253.objecter _send_op 7 to 2.2 on osd.3 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 1 -- 172.21.15.87:0/1947113049 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x7f3e4c006c10 con 0x558a1e7a8830 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:21.749+0000 7f3e72672640 5 client.4253.objecter 1 in flight 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 <== osd.3 v2:172.21.15.87:6816/1100053525 5 ==== osd_op_reply(7 rbd_info [write 0~19] v24'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3e6400a560 con 0x558a1e7a8830 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 10 client.4253.objecter ms_dispatch 0x558a1e5fcac0 osd_op_reply(7 rbd_info [write 0~19] v24'2 uv2 ondisk = 0) 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 10 client.4253.objecter in handle_osd_op_reply 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 7 client.4253.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 10 client.4253.objecter op 0 rval 0 len 0 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 15 client.4253.objecter handle_osd_op_reply completed tid 7 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 15 client.4253.objecter _finish_op 7 2024-07-27T14:41:22.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 4 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 15 client.4253.objecter _session_op_remove 3 7 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 5 client.4253.objecter 0 in flight 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 20 librados: complete_aio_write 0x7f3e50001d60 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 10 librados: watch_flush enter 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 10 librados: watch_flush exit 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 10 client.4253.objecter close_session for osd.3 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x558a1e7a8830 msgr2=0x558a1e7a8c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x558a1e7a8830 0x558a1e7a8c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 20 client.4253.objecter put_session s=0x558a1e71bba0 osd=3 1 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 20 client.4253.objecter shutdown clearing up homeless session... 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 10 client.4253.objecter successfully canceled tick 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3e5406b920 msgr2=0x7f3e5406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3e5406b920 0x7f3e5406dde0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c054000 tx=0x7f3e5c04e000 comp rx=0 tx=0).stop 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 10 monclient: shutdown 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 msgr2=0x558a1e7cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e7600ed80 1 --2- 172.21.15.87:0/1947113049 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x558a1e72abe0 0x558a1e7cc540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e68008530 tx=0x7f3e680089f0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.003+0000 7f3e750d7640 1 -- 172.21.15.87:0/1947113049 reap_dead start 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.004+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 shutdown_connections 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.004+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 >> 172.21.15.87:0/1947113049 conn(0x558a1e6f9a80 msgr2=0x558a1e729200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.004+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 shutdown_connections 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.004+0000 7f3e7600ed80 1 -- 172.21.15.87:0/1947113049 wait complete. 2024-07-27T14:41:22.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.004+0000 7f3e7600ed80 1 librados: shutdown 2024-07-27T14:41:22.008 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-07-27T14:41:22.009 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-27T14:41:22.009 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-07-27T14:41:22.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-07-27T14:41:22.010 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 10 monclient: get_monmap_and_config 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 10 monclient: build_initial_monmap 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 10 monclient: monmap: 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:22.109330+0000 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:22.109330+0000 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:22.109 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 1 Processor -- start 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 1 -- start start 2024-07-27T14:41:22.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.108+0000 7fa425784640 10 monclient: init 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa4200af300 0x7fa4200ac570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): picked mon.noname-c con 0x7fa4200af300 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200ab810 0x7fa420142e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): picked mon.noname-b con 0x7fa4200ab810 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa420143340 0x7fa42013cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): picked mon.noname-a con 0x7fa420143340 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fa4200abd70 con 0x7fa420143340 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fa4200ac050 con 0x7fa4200af300 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa420072eb0 con 0x7fa4200ab810 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.109+0000 7fa425784640 10 monclient(hunting): authenticate will time out at 1222.135680s 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa420143340 0x7fa42013cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41e7fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200ab810 0x7fa420142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41effd640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa4200af300 0x7fa4200ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa420143340 0x7fa42013cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55954/0 (socket says 172.21.15.87:55954) 2024-07-27T14:41:22.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 1 -- 172.21.15.87:0/3033573568 learned_addr learned my addr 172.21.15.87:0/3033573568 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 10 monclient(hunting): get_auth_request con 0x7fa420143340 auth_method 0 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41effd640 10 monclient(hunting): get_auth_request con 0x7fa4200af300 auth_method 0 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41effd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41e7fc640 10 monclient(hunting): get_auth_request con 0x7fa4200ab810 auth_method 0 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41e7fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.110+0000 7fa41e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.111+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 -- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200ab810 msgr2=0x7fa420142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 --2- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200ab810 0x7fa420142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 -- 172.21.15.87:0/3033573568 >> v2:172.21.15.112:3300/0 conn(0x7fa4200af300 msgr2=0x7fa4200ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 --2- 172.21.15.87:0/3033573568 >> v2:172.21.15.112:3300/0 conn(0x7fa4200af300 0x7fa4200ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 monclient: found mon.noname-a 2024-07-27T14:41:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.112+0000 7fa41f7fe640 1 -- 172.21.15.87:0/3033573568 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420073060 con 0x7fa420143340 2024-07-27T14:41:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.114+0000 7fa41f7fe640 1 --2- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa420143340 0x7fa42013cc80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4200b5720 tx=0x7fa41002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fa703459d1adf5 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 1 -- 172.21.15.87:0/3033573568 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41002c9e0 con 0x7fa420143340 2024-07-27T14:41:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: dump: 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:22.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 1 -- 172.21.15.87:0/3033573568 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41002cb80 con 0x7fa420143340 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 1 -- 172.21.15.87:0/3033573568 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4100343f0 con 0x7fa420143340 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa425784640 5 monclient: authenticate success, global_id 4266 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa425784640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa425784640 4 set_mon_vals no callback set 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.118+0000 7fa41dffb640 10 monclient: dump: 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.121 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 10 monclient: shutdown 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 -- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa420143340 msgr2=0x7fa42013cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 --2- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa420143340 0x7fa42013cc80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4200b5720 tx=0x7fa41002be40 comp rx=0 tx=0).stop 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 -- 172.21.15.87:0/3033573568 shutdown_connections 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 --2- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa420143340 0x7fa42013cc80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 --2- 172.21.15.87:0/3033573568 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200ab810 0x7fa420142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 --2- 172.21.15.87:0/3033573568 >> v2:172.21.15.112:3300/0 conn(0x7fa4200af300 0x7fa4200ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 -- 172.21.15.87:0/3033573568 >> 172.21.15.87:0/3033573568 conn(0x7fa420067f60 msgr2=0x7fa4200b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 -- 172.21.15.87:0/3033573568 shutdown_connections 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 1 -- 172.21.15.87:0/3033573568 wait complete. 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 10 monclient: build_initial_monmap 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.119+0000 7fa425784640 10 monclient: monmap: 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 librados: starting msgr at 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 librados: starting objecter 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 Processor -- start 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 -- start start 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 librados: setting wanted keys 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 librados: calling monclient init 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient: init 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:22.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): picked mon.noname-a con 0x7fa4200ab810 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200af300 0x7fa4201d3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): picked mon.noname-c con 0x7fa4200af300 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa420143340 0x7fa420139700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): picked mon.noname-b con 0x7fa420143340 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa4200af920 con 0x7fa4200ab810 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa42013d410 con 0x7fa420143340 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa4200afaa0 con 0x7fa4200af300 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:22.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.120+0000 7fa425784640 10 monclient(hunting): authenticate will time out at 1222.146946s 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41e7fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200af300 0x7fa4201d3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa420143340 0x7fa420139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55962/0 (socket says 172.21.15.87:55962) 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 1 -- 172.21.15.87:0/1588635175 learned_addr learned my addr 172.21.15.87:0/1588635175 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 10 monclient(hunting): get_auth_request con 0x7fa4200ab810 auth_method 0 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41e7fc640 10 monclient(hunting): get_auth_request con 0x7fa4200af300 auth_method 0 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41e7fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41f7fe640 10 monclient(hunting): get_auth_request con 0x7fa420143340 auth_method 0 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.121+0000 7fa41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.122+0000 7fa41e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-07-27T14:41:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 -- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200af300 msgr2=0x7fa4201d3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200af300 0x7fa4201d3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 -- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa420143340 msgr2=0x7fa420139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa420143340 0x7fa420139700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 monclient: found mon.noname-a 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.124+0000 7fa41effd640 1 -- 172.21.15.87:0/1588635175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420071f30 con 0x7fa4200ab810 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.127+0000 7fa41effd640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa41800ea10 tx=0x7fa41800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.127+0000 7fa4177fe640 10 client.?.objecter ms_handle_connect 0x7fa4200ab810 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.127+0000 7fa4177fe640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418017070 con 0x7fa4200ab810 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: dump: 2024-07-27T14:41:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41800ca00 con 0x7fa4200ab810 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41800cdf0 con 0x7fa4200ab810 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa4177fe640 10 monclient: dump: 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 5 monclient: authenticate success, global_id 4269 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 10 monclient: _renew_subs 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa41d7fa640 4 set_mon_vals no callback set 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa420072350 con 0x7fa4200ab810 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 10 monclient: _renew_subs 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa420139e20 con 0x7fa4200ab810 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.130+0000 7fa425784640 1 librados: init done 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.131+0000 7fa425784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.131+0000 7fa425784640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:22.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.131+0000 7fa425784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.131+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa420071d60 con 0x7fa4200ab810 2024-07-27T14:41:22.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.149+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa41801a070 con 0x7fa4200ab810 2024-07-27T14:41:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.150+0000 7fa4177fe640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 0x7fa40006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.150+0000 7fa41e7fc640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 0x7fa40006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.150+0000 7fa41e7fc640 10 monclient: get_auth_request con 0x7fa40006d810 auth_method 0 2024-07-27T14:41:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.150+0000 7fa41e7fc640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 0x7fa40006fcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4200718a0 tx=0x7fa408009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.150+0000 7fa4177fe640 10 client.4269.objecter ms_handle_connect 0x7fa40006d810 2024-07-27T14:41:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.152+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7fa418057a80 con 0x7fa4200ab810 2024-07-27T14:41:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.152+0000 7fa4177fe640 10 client.4269.objecter ms_dispatch 0x7fa420000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.152+0000 7fa4177fe640 3 client.4269.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.152+0000 7fa4177fe640 3 client.4269.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.152+0000 7fa4177fe640 20 client.4269.objecter dump_active .. 0 homeless 2024-07-27T14:41:22.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.389+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fa4180599e0 con 0x7fa4200ab810 2024-07-27T14:41:22.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.389+0000 7fa4177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:22.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.389+0000 7fa4177fe640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:22.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.616+0000 7fa425784640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-07-27T14:41:22.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.616+0000 7fa425784640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-07-27T14:41:22.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.616+0000 7fa425784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.616+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa42013a140 con 0x7fa4200ab810 2024-07-27T14:41:22.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.747+0000 7fa4177fe640 1 -- 172.21.15.87:0/1588635175 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172946 (secure 0 0 0) 0x7fa41805dce0 con 0x7fa4200ab810 2024-07-27T14:41:22.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.747+0000 7fa4177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-07-27T14:41:22.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.747+0000 7fa4177fe640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:22.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 10 librados: watch_flush enter 2024-07-27T14:41:22.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 10 librados: watch_flush exit 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 20 client.4269.objecter shutdown clearing up homeless session... 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 10 client.4269.objecter successfully canceled tick 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 msgr2=0x7fa40006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 0x7fa40006fcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa4200718a0 tx=0x7fa408009190 comp rx=0 tx=0).stop 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 10 monclient: shutdown 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 msgr2=0x7fa42013f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.749+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa41800ea10 tx=0x7fa41800eed0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 shutdown_connections 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa40006d810 0x7fa40006fcd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa420143340 0x7fa420139700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa4200af300 0x7fa4201d3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 --2- 172.21.15.87:0/1588635175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa4200ab810 0x7fa42013f870 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 >> 172.21.15.87:0/1588635175 conn(0x7fa420067f60 msgr2=0x7fa4200b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:22.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 shutdown_connections 2024-07-27T14:41:22.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 -- 172.21.15.87:0/1588635175 wait complete. 2024-07-27T14:41:22.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.750+0000 7fa425784640 1 librados: shutdown 2024-07-27T14:41:22.762 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1547416426},{"type":"v1","addr":"172.21.15.112:6801","nonce":1547416426}]},"active_addr":"172.21.15.112:6801/1547416426","active_change":"2024-07-27T14:40:19.552519+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4115488588}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3183114139}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2636405169}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2901301380}]}]} 2024-07-27T14:41:22.767 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-07-27T14:41:22.767 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-07-27T14:41:22.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-27T14:41:22.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.869+0000 7fec9ef6a640 10 monclient: get_monmap_and_config 2024-07-27T14:41:22.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.869+0000 7fec9ef6a640 10 monclient: build_initial_monmap 2024-07-27T14:41:22.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.869+0000 7fec9ef6a640 10 monclient: monmap: 2024-07-27T14:41:22.870 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:22.870 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:22.870910+0000 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:22.870910+0000 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:22.871 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.870+0000 7fec9ef6a640 1 Processor -- start 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.870+0000 7fec9ef6a640 1 -- start start 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.870+0000 7fec9ef6a640 10 monclient: init 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fec980b5aa0 0x7fec980aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-a con 0x7fec980b5aa0 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:22.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec980b2e30 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b4d80 0x7fec9813ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-b con 0x7fec980b4d80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec980b61b0 con 0x7fec980b2e30 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fec980b5e80 con 0x7fec980b5aa0 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fec980b48f0 con 0x7fec980b4d80 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9ef6a640 10 monclient(hunting): authenticate will time out at 1222.897309s 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.871+0000 7fec9cd06640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fec980b5aa0 0x7fec980aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fec980b5aa0 0x7fec980aaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42612/0 (socket says 172.21.15.87:42612) 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 1 -- 172.21.15.87:0/1802329786 learned_addr learned my addr 172.21.15.87:0/1802329786 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): get_auth_request con 0x7fec980b5aa0 auth_method 0 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.872+0000 7fec9cd06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-07-27T14:41:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 1 -- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b4d80 msgr2=0x7fec9813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 1 --2- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b4d80 0x7fec9813ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 1 -- 172.21.15.87:0/1802329786 >> v1:172.21.15.112:6789/0 conn(0x7fec980b2e30 legacy=0x7fec98146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 10 monclient: _finish_hunting 0 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 1 monclient: found mon.noname-a 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.874+0000 7fec9cd06640 1 -- 172.21.15.87:0/1802329786 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec980b5210 con 0x7fec980b5aa0 2024-07-27T14:41:22.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.876+0000 7fec9cd06640 1 --2- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b5aa0 0x7fec980aaa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45fb91ebf69783c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 1 -- 172.21.15.87:0/1802329786 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c038070 con 0x7fec980b5aa0 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: dump: 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.880 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 1 -- 172.21.15.87:0/1802329786 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8c031740 con 0x7fec980b5aa0 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec9ef6a640 5 monclient: authenticate success, global_id 4272 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec9ef6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec9ef6a640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec9ef6a640 4 set_mon_vals no callback set 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 1 -- 172.21.15.87:0/1802329786 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c02f050 con 0x7fec980b5aa0 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.879+0000 7fec977fe640 10 monclient: dump: 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.883 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.883 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.883 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 10 monclient: shutdown 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 -- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b5aa0 msgr2=0x7fec980aaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b5aa0 0x7fec980aaa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02b7e0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 -- 172.21.15.87:0/1802329786 shutdown_connections 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b4d80 0x7fec9813ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/1802329786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b5aa0 0x7fec980aaa50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 -- 172.21.15.87:0/1802329786 >> 172.21.15.87:0/1802329786 conn(0x7fec98067810 msgr2=0x7fec98067c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 -- 172.21.15.87:0/1802329786 shutdown_connections 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.880+0000 7fec9ef6a640 1 -- 172.21.15.87:0/1802329786 wait complete. 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 10 monclient: build_initial_monmap 2024-07-27T14:41:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 10 monclient: monmap: 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 librados: starting msgr at 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 librados: starting objecter 2024-07-27T14:41:22.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 Processor -- start 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 -- start start 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 librados: setting wanted keys 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 1 librados: calling monclient init 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.881+0000 7fec9ef6a640 10 monclient: init 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b2e30 0x7fec98147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-c con 0x7fec980b2e30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-a con 0x7fec980b4d80 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec980b5aa0 0x7fec9814ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): picked mon.noname-b con 0x7fec980b5aa0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fec981408f0 con 0x7fec980b4d80 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fec98140770 con 0x7fec980b5aa0 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fec9814d4e0 con 0x7fec980b2e30 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec9ef6a640 10 monclient(hunting): authenticate will time out at 1222.908253s 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42620/0 (socket says 172.21.15.87:42620) 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 1 -- 172.21.15.87:0/2354033073 learned_addr learned my addr 172.21.15.87:0/2354033073 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 10 monclient(hunting): get_auth_request con 0x7fec980b4d80 auth_method 0 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.882+0000 7fec97fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:22.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.883+0000 7fec97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:22.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.883+0000 7fec97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:22.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.883+0000 7fec97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:22.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 -- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b2e30 msgr2=0x7fec98147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b2e30 0x7fec98147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 -- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec980b5aa0 msgr2=0x7fec9814ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec980b5aa0 0x7fec9814ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 monclient: found mon.noname-a 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.886+0000 7fec97fff640 1 -- 172.21.15.87:0/2354033073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec98149e10 con 0x7fec980b4d80 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.888+0000 7fec97fff640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c410 tx=0x7fec8400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.888+0000 7fec957fa640 10 client.?.objecter ms_handle_connect 0x7fec980b4d80 2024-07-27T14:41:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.888+0000 7fec957fa640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84017070 con 0x7fec980b4d80 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: dump: 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.893 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: _finish_auth 0 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: _check_auth_tickets 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec840035f0 con 0x7fec980b4d80 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec840039e0 con 0x7fec980b4d80 2024-07-27T14:41:22.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 5 monclient: authenticate success, global_id 4275 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 10 monclient: _renew_subs 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9814a0f0 con 0x7fec980b4d80 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec96ffd640 4 set_mon_vals no callback set 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec957fa640 10 monclient: dump: 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:22.895 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 10 monclient: _renew_subs 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec98148710 con 0x7fec980b4d80 2024-07-27T14:41:22.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.892+0000 7fec9ef6a640 1 librados: init done 2024-07-27T14:41:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.893+0000 7fec9ef6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.893+0000 7fec9ef6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.893+0000 7fec9ef6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:22.893+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec981491f0 con 0x7fec980b4d80 2024-07-27T14:41:23.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.140+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fec8401a070 con 0x7fec980b4d80 2024-07-27T14:41:23.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.140+0000 7fec957fa640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 0x7fec7c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.140+0000 7fec9cd06640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 0x7fec7c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.140+0000 7fec9cd06640 10 monclient: get_auth_request con 0x7fec7c06d840 auth_method 0 2024-07-27T14:41:23.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.141+0000 7fec9cd06640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 0x7fec7c06fd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec8c035000 tx=0x7fec8c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:23.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.141+0000 7fec957fa640 10 client.4275.objecter ms_handle_connect 0x7fec7c06d840 2024-07-27T14:41:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7fec84056a70 con 0x7fec980b4d80 2024-07-27T14:41:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 10 client.4275.objecter ms_dispatch 0x7fec98000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 3 client.4275.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 3 client.4275.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 20 client.4275.objecter dump_active .. 0 homeless 2024-07-27T14:41:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fec84021020 con 0x7fec980b4d80 2024-07-27T14:41:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.154+0000 7fec957fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:23.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.340+0000 7fec9ef6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:23.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.340+0000 7fec9ef6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:23.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.340+0000 7fec9ef6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.340+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec9813ffa0 con 0x7fec980b4d80 2024-07-27T14:41:23.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.375+0000 7fec957fa640 1 -- 172.21.15.87:0/2354033073 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+14825 (secure 0 0 0) 0x7fec840590d0 con 0x7fec980b4d80 2024-07-27T14:41:23.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.375+0000 7fec957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:23.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.375+0000 7fec957fa640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:23.377 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:23.377 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":24,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:41:21.582189+0000","last_up_change":"2024-07-27T14:41:10.123887+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T14:41:00.102594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-27T14:41:16.750108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6809","nonce":3385608780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6811","nonce":3385608780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6815","nonce":3385608780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6813","nonce":3385608780}]},"public_addr":"172.21.15.87:6809/3385608780","cluster_addr":"172.21.15.87:6811/3385608780","heartbeat_back_addr":"172.21.15.87:6815/3385608780","heartbeat_front_addr":"172.21.15.87:6813/3385608780","state":["exists","up"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6801","nonce":1509783986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6803","nonce":1509783986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6807","nonce":1509783986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6805","nonce":1509783986}]},"public_addr":"172.21.15.87:6801/1509783986","cluster_addr":"172.21.15.87:6803/1509783986","heartbeat_back_addr":"172.21.15.87:6807/1509783986","heartbeat_front_addr":"172.21.15.87:6805/1509783986","state":["exists","up"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6822","nonce":357294290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6825","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6827","nonce":357294290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6831","nonce":357294290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6829","nonce":357294290}]},"public_addr":"172.21.15.87:6822/357294290","cluster_addr":"172.21.15.87:6827/357294290","heartbeat_back_addr":"172.21.15.87:6831/357294290","heartbeat_front_addr":"172.21.15.87:6829/357294290","state":["exists","up"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6817","nonce":1100053525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6819","nonce":1100053525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6826","nonce":1100053525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6823","nonce":1100053525}]},"public_addr":"172.21.15.87:6817/1100053525","cluster_addr":"172.21.15.87:6819/1100053525","heartbeat_back_addr":"172.21.15.87:6826/1100053525","heartbeat_front_addr":"172.21.15.87:6823/1100053525","state":["exists","up"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6803","nonce":591715754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6805","nonce":591715754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6809","nonce":591715754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6807","nonce":591715754}]},"public_addr":"172.21.15.112:6803/591715754","cluster_addr":"172.21.15.112:6805/591715754","heartbeat_back_addr":"172.21.15.112:6809/591715754","heartbeat_front_addr":"172.21.15.112:6807/591715754","state":["exists","up"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.359413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.864798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.468303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.219576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.792415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.790276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.126847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.527378+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:41:23.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 10 librados: watch_flush enter 2024-07-27T14:41:23.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 10 librados: watch_flush exit 2024-07-27T14:41:23.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 20 client.4275.objecter shutdown clearing up homeless session... 2024-07-27T14:41:23.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 10 client.4275.objecter successfully canceled tick 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 msgr2=0x7fec7c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 0x7fec7c06fd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec8c035000 tx=0x7fec8c030000 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 10 monclient: shutdown 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 msgr2=0x7fec9814a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c410 tx=0x7fec8400c8d0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 shutdown_connections 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fec7c06d840 0x7fec7c06fd00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec980b5aa0 0x7fec9814ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fec980b4d80 0x7fec9814a7f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 --2- 172.21.15.87:0/2354033073 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fec980b2e30 0x7fec98147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 >> 172.21.15.87:0/2354033073 conn(0x7fec98067810 msgr2=0x7fec9814c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 shutdown_connections 2024-07-27T14:41:23.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 -- 172.21.15.87:0/2354033073 wait complete. 2024-07-27T14:41:23.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.378+0000 7fec9ef6a640 1 librados: shutdown 2024-07-27T14:41:23.389 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-07-27T14:41:23.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.488+0000 7f350a974640 10 monclient: get_monmap_and_config 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.488+0000 7f350a974640 10 monclient: build_initial_monmap 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.488+0000 7f350a974640 10 monclient: monmap: 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:23.489875+0000 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:23.489875+0000 2024-07-27T14:41:23.489 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:23.490 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.489+0000 7f350a974640 1 Processor -- start 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.489+0000 7f350a974640 1 -- start start 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.489+0000 7f350a974640 10 monclient: init 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f350406bbb0 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3504146d50 0x7f350406bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): picked mon.noname-c con 0x7f3504146d50 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f350413fd90 0x7f3504140220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): picked mon.noname-a con 0x7f350413fd90 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350406d840 con 0x7f350406bbb0 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f35041461b0 con 0x7f350413fd90 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f350406c7b0 con 0x7f3504146d50 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f350a974640 10 monclient(hunting): authenticate will time out at 1223.516473s 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.490+0000 7f3508f11640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f350413fd90 0x7f3504140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f35037fe640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3504146d50 0x7f350406bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f350413fd90 0x7f3504140220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42634/0 (socket says 172.21.15.87:42634) 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 1 -- 172.21.15.87:0/3637752162 learned_addr learned my addr 172.21.15.87:0/3637752162 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:23.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): get_auth_request con 0x7f350413fd90 auth_method 0 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f35037fe640 10 monclient(hunting): get_auth_request con 0x7f3504146d50 auth_method 0 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f35037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f35037fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f35037fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:23.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.491+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.492+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.492+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.492+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 1 -- 172.21.15.87:0/3637752162 >> v2:172.21.15.112:3300/0 conn(0x7f3504146d50 msgr2=0x7f350406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 1 --2- 172.21.15.87:0/3637752162 >> v2:172.21.15.112:3300/0 conn(0x7f3504146d50 0x7f350406bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 1 -- 172.21.15.87:0/3637752162 >> v1:172.21.15.112:6789/0 conn(0x7f350406bbb0 legacy=0x7f3504146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 10 monclient: _finish_hunting 0 2024-07-27T14:41:23.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 1 monclient: found mon.noname-a 2024-07-27T14:41:23.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:23.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.493+0000 7f3508f11640 1 -- 172.21.15.87:0/3637752162 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3504140760 con 0x7f350413fd90 2024-07-27T14:41:23.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.495+0000 7f3508f11640 1 --2- 172.21.15.87:0/3637752162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f350413fd90 0x7f3504140220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3504146330 tx=0x7f34fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f1b12d07890ca server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:23.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 1 -- 172.21.15.87:0/3637752162 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc02cb00 con 0x7f350413fd90 2024-07-27T14:41:23.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:23.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:23.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: dump: 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:23.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 1 -- 172.21.15.87:0/3637752162 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc02cca0 con 0x7f350413fd90 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 1 -- 172.21.15.87:0/3637752162 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc034610 con 0x7f350413fd90 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f350a974640 5 monclient: authenticate success, global_id 4278 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f350a974640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f350a974640 4 set_mon_vals no callback set 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.499+0000 7f3502ffd640 10 monclient: dump: 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.501 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:23.502 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.502 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:23.502 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:23.502 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:23.502 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 10 monclient: shutdown 2024-07-27T14:41:23.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 -- 172.21.15.87:0/3637752162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f350413fd90 msgr2=0x7f3504140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 --2- 172.21.15.87:0/3637752162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f350413fd90 0x7f3504140220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3504146330 tx=0x7f34fc02bf60 comp rx=0 tx=0).stop 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 -- 172.21.15.87:0/3637752162 shutdown_connections 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 --2- 172.21.15.87:0/3637752162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f350413fd90 0x7f3504140220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 --2- 172.21.15.87:0/3637752162 >> v2:172.21.15.112:3300/0 conn(0x7f3504146d50 0x7f350406bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 -- 172.21.15.87:0/3637752162 >> 172.21.15.87:0/3637752162 conn(0x7f3504067ee0 msgr2=0x7f35040b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 -- 172.21.15.87:0/3637752162 shutdown_connections 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 1 -- 172.21.15.87:0/3637752162 wait complete. 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 10 monclient: build_initial_monmap 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.500+0000 7f350a974640 10 monclient: monmap: 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:23.506 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 librados: starting msgr at 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 librados: starting objecter 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 Processor -- start 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 -- start start 2024-07-27T14:41:23.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 librados: setting wanted keys 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 librados: calling monclient init 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient: init 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350406bbb0 0x7f3504072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): picked mon.noname-b con 0x7f350406bbb0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f350413fd90 0x7f3504142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): picked mon.noname-c con 0x7f350413fd90 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): picked mon.noname-a con 0x7f3504146d50 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f35040af430 con 0x7f3504146d50 2024-07-27T14:41:23.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35040af2b0 con 0x7f350406bbb0 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.501+0000 7f350a974640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f350414d490 con 0x7f350413fd90 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f350a974640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f350a974640 10 monclient(hunting): authenticate will time out at 1223.528014s 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f35037fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f350413fd90 0x7f3504142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3503fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350406bbb0 0x7f3504072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42636/0 (socket says 172.21.15.87:42636) 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 1 -- 172.21.15.87:0/4080340860 learned_addr learned my addr 172.21.15.87:0/4080340860 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 10 monclient(hunting): get_auth_request con 0x7f3504146d50 auth_method 0 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:23.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f3508f11640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f35037fe640 10 monclient(hunting): get_auth_request con 0x7f350413fd90 auth_method 0 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f35037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f35037fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.502+0000 7f35037fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): get_auth_request con 0x7f350406bbb0 auth_method 0 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3508f11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:23.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:23.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f35037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:23.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:23.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:23.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.503+0000 7f3503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:23.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 -- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f350413fd90 msgr2=0x7f3504142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f350413fd90 0x7f3504142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 -- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350406bbb0 msgr2=0x7f3504072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350406bbb0 0x7f3504072e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 10 monclient: _finish_hunting 0 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 monclient: found mon.noname-a 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.506+0000 7f3508f11640 1 -- 172.21.15.87:0/4080340860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3504143130 con 0x7f3504146d50 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.508+0000 7f3508f11640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036040 tx=0x7f34fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.508+0000 7f3500ff9640 10 client.?.objecter ms_handle_connect 0x7f3504146d50 2024-07-27T14:41:23.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.508+0000 7f3500ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc043020 con 0x7f3504146d50 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: dump: 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.514 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc02ce50 con 0x7f3504146d50 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 5 monclient: authenticate success, global_id 4281 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 10 monclient: _renew_subs 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35041434c0 con 0x7f3504146d50 2024-07-27T14:41:23.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc03e690 con 0x7f3504146d50 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f3500ff9640 10 monclient: dump: 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:23.516 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 10 monclient: _renew_subs 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350413d580 con 0x7f3504146d50 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f350a974640 1 librados: init done 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.512+0000 7f35027fc640 4 set_mon_vals no callback set 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.513+0000 7f350a974640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.513+0000 7f350a974640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.513+0000 7f350a974640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.513+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3504140220 con 0x7f3504146d50 2024-07-27T14:41:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.763+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f34fc046020 con 0x7f3504146d50 2024-07-27T14:41:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.763+0000 7f3500ff9640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 0x7f34f006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.764+0000 7f3503fff640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 0x7f34f006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.764+0000 7f3503fff640 10 monclient: get_auth_request con 0x7f34f006d810 auth_method 0 2024-07-27T14:41:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.764+0000 7f3500ff9640 10 client.4281.objecter ms_handle_connect 0x7f34f006d810 2024-07-27T14:41:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.764+0000 7f3503fff640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 0x7f34f006fcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000f60 tx=0x7f34f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:23.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f34fc079c30 con 0x7f3504146d50 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 10 client.4281.objecter ms_dispatch 0x7f3504000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 3 client.4281.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 3 client.4281.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 20 client.4281.objecter dump_active .. 0 homeless 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f34fc04d030 con 0x7f3504146d50 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.778+0000 7f3500ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:23.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.975+0000 7f350a974640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:23.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.975+0000 7f350a974640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:23.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.975+0000 7f350a974640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:23.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:23.975+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35040b4fc0 con 0x7f3504146d50 2024-07-27T14:41:24.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.002+0000 7f3500ff9640 1 -- 172.21.15.87:0/4080340860 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+14825 (secure 0 0 0) 0x7f34fc07bb90 con 0x7f3504146d50 2024-07-27T14:41:24.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.002+0000 7f3500ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T14:41:24.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.002+0000 7f3500ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:24.003 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:24.004 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":24,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T14:41:21.582189+0000","last_up_change":"2024-07-27T14:41:10.123887+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T14:41:00.102594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-27T14:41:16.750108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6809","nonce":3385608780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6811","nonce":3385608780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6815","nonce":3385608780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6813","nonce":3385608780}]},"public_addr":"172.21.15.87:6809/3385608780","cluster_addr":"172.21.15.87:6811/3385608780","heartbeat_back_addr":"172.21.15.87:6815/3385608780","heartbeat_front_addr":"172.21.15.87:6813/3385608780","state":["exists","up"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6801","nonce":1509783986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6803","nonce":1509783986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6807","nonce":1509783986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6805","nonce":1509783986}]},"public_addr":"172.21.15.87:6801/1509783986","cluster_addr":"172.21.15.87:6803/1509783986","heartbeat_back_addr":"172.21.15.87:6807/1509783986","heartbeat_front_addr":"172.21.15.87:6805/1509783986","state":["exists","up"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6822","nonce":357294290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6825","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6827","nonce":357294290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6831","nonce":357294290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6829","nonce":357294290}]},"public_addr":"172.21.15.87:6822/357294290","cluster_addr":"172.21.15.87:6827/357294290","heartbeat_back_addr":"172.21.15.87:6831/357294290","heartbeat_front_addr":"172.21.15.87:6829/357294290","state":["exists","up"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6817","nonce":1100053525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6819","nonce":1100053525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6826","nonce":1100053525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6823","nonce":1100053525}]},"public_addr":"172.21.15.87:6817/1100053525","cluster_addr":"172.21.15.87:6819/1100053525","heartbeat_back_addr":"172.21.15.87:6826/1100053525","heartbeat_front_addr":"172.21.15.87:6823/1100053525","state":["exists","up"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6803","nonce":591715754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6805","nonce":591715754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6809","nonce":591715754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6807","nonce":591715754}]},"public_addr":"172.21.15.112:6803/591715754","cluster_addr":"172.21.15.112:6805/591715754","heartbeat_back_addr":"172.21.15.112:6809/591715754","heartbeat_front_addr":"172.21.15.112:6807/591715754","state":["exists","up"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.359413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.864798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.468303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.219576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.792415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.790276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.126847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.527378+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T14:41:24.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.004+0000 7f350a974640 10 librados: watch_flush enter 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.004+0000 7f350a974640 10 librados: watch_flush exit 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.004+0000 7f350a974640 20 client.4281.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.004+0000 7f350a974640 10 client.4281.objecter successfully canceled tick 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 msgr2=0x7f34f006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 0x7f34f006fcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000f60 tx=0x7f34f4009040 comp rx=0 tx=0).stop 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 10 monclient: shutdown 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 msgr2=0x7f35041d8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036040 tx=0x7f34fc004a40 comp rx=0 tx=0).stop 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 shutdown_connections 2024-07-27T14:41:24.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f34f006d810 0x7f34f006fcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3504146d50 0x7f35041d8680 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f350413fd90 0x7f3504142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 --2- 172.21.15.87:0/4080340860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f350406bbb0 0x7f3504072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 >> 172.21.15.87:0/4080340860 conn(0x7f3504067ee0 msgr2=0x7f35040b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 shutdown_connections 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 -- 172.21.15.87:0/4080340860 wait complete. 2024-07-27T14:41:24.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.005+0000 7f350a974640 1 librados: shutdown 2024-07-27T14:41:24.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-07-27T14:41:24.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-07-27T14:41:24.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-07-27T14:41:24.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-07-27T14:41:24.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-07-27T14:41:24.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-07-27T14:41:24.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-07-27T14:41:24.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-07-27T14:41:24.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.189+0000 7f9bb041a640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.189+0000 7f9bb041a640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.190+0000 7f9bb041a640 10 monclient: monmap: 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.191034+0000 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.191034+0000 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.191 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.192 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.192 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.190+0000 7f3f8359e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.190+0000 7f3f8359e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.190+0000 7f3f8359e640 10 monclient: monmap: 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.191190+0000 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.191190+0000 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.193 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 1 Processor -- start 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 1 -- start start 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 10 monclient: init 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.192+0000 7f3f8359e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 0x7f3f840aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f9bb041a640 1 Processor -- start 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f9bb041a640 1 -- start start 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f9bb041a640 10 monclient: init 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840b19f0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f840ab2e0 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f840b0720 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840ad960 con 0x7f3f840b0720 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840b1dd0 con 0x7f3f840ab2e0 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f3f840abb80 con 0x7f3f840b19f0 2024-07-27T14:41:24.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f8359e640 10 monclient(hunting): authenticate will time out at 1224.219292s 2024-07-27T14:41:24.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f82d9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3f840b0720 0x7f3f8415b490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45208/0 (socket says 172.21.15.87:45208) 2024-07-27T14:41:24.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.193+0000 7f3f82d9d640 1 -- 172.21.15.87:0/2786497930 learned_addr learned my addr 172.21.15.87:0/2786497930 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-07-27T14:41:24.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ba80abe20 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ba80ac200 0x7f9ba80ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba80ac200 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 0x7f9ba81442d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba80734a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ba80aec70 con 0x7f9ba80abe20 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f9ba80ae0a0 con 0x7f9ba80ac200 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba8144810 con 0x7f9ba80734a0 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.194+0000 7f9bb041a640 10 monclient(hunting): authenticate will time out at 1224.220336s 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.195+0000 7f244f59e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.195+0000 7f244f59e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.195+0000 7f244f59e640 10 monclient: monmap: 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.196674+0000 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.196674+0000 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.199 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.196+0000 7f9bae1b6640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f9ba80abe20 0x7f9ba80ad990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45218/0 (socket says 172.21.15.87:45218) 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.196+0000 7f9bae1b6640 1 -- 172.21.15.87:0/4209712603 learned_addr learned my addr 172.21.15.87:0/4209712603 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.196+0000 7f244f59e640 1 Processor -- start 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.197+0000 7f244f59e640 1 -- start start 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.197+0000 7f244f59e640 10 monclient: init 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f245013bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f24500aa9b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f24500aad90 0x7f245013c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f24500aad90 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f245013ed50 0x7f245013f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f245013ed50 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f24500abd60 con 0x7f245013ed50 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f24500ace80 con 0x7f24500aad90 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f24500ab4c0 con 0x7f24500aa9b0 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f244f59e640 10 monclient(hunting): authenticate will time out at 1224.224194s 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7ff9de862640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7ff9de862640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7ff9de862640 10 monclient: monmap: 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.199875+0000 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.199875+0000 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.202 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f9bae9b7640 1 --2- 172.21.15.87:0/4209712603 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 0x7f9ba81442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f9bae9b7640 10 monclient(hunting): get_auth_request con 0x7f9ba80734a0 auth_method 0 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f9bae9b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f9bae9b7640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.198+0000 7f9bae9b7640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.199+0000 7f9bae9b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.199+0000 7f9bae9b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.199+0000 7f9bae9b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.200+0000 7ff9de862640 1 Processor -- start 2024-07-27T14:41:24.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.200+0000 7ff9de862640 1 -- start start 2024-07-27T14:41:24.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.200+0000 7ff9de862640 10 monclient: init 2024-07-27T14:41:24.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ff9d80b1b80 0x7ff9d80aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d80b1b80 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9d80ad460 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9d8142cb0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d8136610 con 0x7ff9d8142cb0 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d8073f00 con 0x7ff9d80ad460 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d80ae6a0 con 0x7ff9d80b1b80 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.201+0000 7ff9de862640 10 monclient(hunting): authenticate will time out at 1224.227900s 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 1 -- 172.21.15.87:0/4209712603 >> v2:172.21.15.112:3300/0 conn(0x7f9ba80ac200 msgr2=0x7f9ba80ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 1 --2- 172.21.15.87:0/4209712603 >> v2:172.21.15.112:3300/0 conn(0x7f9ba80ac200 0x7f9ba80ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 1 -- 172.21.15.87:0/4209712603 >> v1:172.21.15.87:6789/0 conn(0x7f9ba80abe20 legacy=0x7f9ba80ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.202+0000 7f9bae9b7640 1 -- 172.21.15.87:0/4209712603 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba81449c0 con 0x7f9ba80734a0 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.203+0000 7f631def0640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.203+0000 7f631def0640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.203+0000 7f631def0640 10 monclient: monmap: 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.204731+0000 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.204731+0000 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.208 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f245013bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f245013bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39146/0 (socket says 172.21.15.87:39146) 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 1 -- 172.21.15.87:0/3637172810 learned_addr learned my addr 172.21.15.87:0/3637172810 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 10 monclient(hunting): get_auth_request con 0x7f24500aa9b0 auth_method 0 2024-07-27T14:41:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f244e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f1e2ca0e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f1e2ca0e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.204+0000 7f1e2ca0e640 10 monclient: monmap: 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.205904+0000 2024-07-27T14:41:24.210 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.205904+0000 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f631def0640 1 Processor -- start 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f631def0640 1 -- start start 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f631def0640 10 monclient: init 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f27c0c4d640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f27c0c4d640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f27c0c4d640 10 monclient: monmap: 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.206397+0000 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.206397+0000 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.212 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f1e2ca0e640 1 Processor -- start 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f244e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f244e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f244e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f1e2ca0e640 1 -- start start 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.205+0000 7f1e2ca0e640 10 monclient: init 2024-07-27T14:41:24.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-07-27T14:41:24.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f27c0c4d640 1 Processor -- start 2024-07-27T14:41:24.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f63180b1a80 0x7f631813bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): picked mon.noname-c con 0x7f63180b1a80 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63180ac340 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 0x7f63180ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f27c0c4d640 1 -- start start 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): picked mon.noname-a con 0x7f631813c320 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6318073db0 con 0x7f63180ac340 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f63180b1e60 con 0x7f631813c320 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f63180ad480 con 0x7f63180b1a80 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f631def0640 10 monclient(hunting): authenticate will time out at 1224.232997s 2024-07-27T14:41:24.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-07-27T14:41:24.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f1e28072b70 0x7f1e280b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.206+0000 7f27c0c4d640 10 monclient: init 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-a con 0x7f1e28072b70 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e280b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-b con 0x7f1e280af970 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e280b1c80 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e280aff60 con 0x7f1e280b1c80 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f1e28072f50 con 0x7f1e28072b70 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1e280b3060 con 0x7f1e280af970 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f1e2ca0e640 10 monclient(hunting): authenticate will time out at 1224.233263s 2024-07-27T14:41:24.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f9bae9b7640 1 --2- 172.21.15.87:0/4209712603 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 0x7f9ba81442d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ce37624bf3ce4d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f3f8259c640 1 --2- 172.21.15.87:0/2786497930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 0x7f3f840aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f27b4006bf0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f27b402be60 0x7f27b402f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.207+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f27b402be60 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27b40315a0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b4032c50 con 0x7f27b40315a0 2024-07-27T14:41:24.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f27b4006fd0 con 0x7f27b402be60 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f27b402f6d0 con 0x7f27b4006bf0 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f27c0c4d640 10 monclient(hunting): authenticate will time out at 1224.234106s 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f3f8259c640 10 monclient(hunting): get_auth_request con 0x7f3f840b19f0 auth_method 0 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f3f8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f3f8259c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.208+0000 7f3f8259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.210+0000 7f244ed9d640 1 --2- 172.21.15.87:0/3637172810 >> v2:172.21.15.87:3300/0 conn(0x7f245013ed50 0x7f245013f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.212+0000 7f631d6ef640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 0x7f63180ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.214+0000 7faa7a3a0640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.214+0000 7faa7a3a0640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.214+0000 7faa7a3a0640 10 monclient: monmap: 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.215952+0000 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.215952+0000 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.225 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.215+0000 7faa7a3a0640 1 Processor -- start 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.215+0000 7faa7a3a0640 1 -- start start 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.215+0000 7faa7a3a0640 10 monclient: init 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa740728e0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa740b4a50 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7faa740aa890 0x7faa7414e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-a con 0x7faa740aa890 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa740b4530 con 0x7faa740728e0 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa740aace0 con 0x7faa740b4a50 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7faa74072cc0 con 0x7faa740aa890 2024-07-27T14:41:24.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.216+0000 7faa7a3a0640 10 monclient(hunting): authenticate will time out at 1224.242665s 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.232+0000 7f244ed9d640 10 monclient(hunting): get_auth_request con 0x7f245013ed50 auth_method 0 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.232+0000 7f244ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.232+0000 7f244ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.232+0000 7f244ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 0x7f63180ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42676/0 (socket says 172.21.15.87:42676) 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 1 -- 172.21.15.87:0/630505457 learned_addr learned my addr 172.21.15.87:0/630505457 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 10 monclient(hunting): get_auth_request con 0x7f631813c320 auth_method 0 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.233+0000 7f631d6ef640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.235+0000 7f244dd9b640 1 --2- 172.21.15.87:0/3637172810 >> v2:172.21.15.112:3300/0 conn(0x7f24500aad90 0x7f245013c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.236+0000 7f244dd9b640 10 monclient(hunting): get_auth_request con 0x7f24500aad90 auth_method 0 2024-07-27T14:41:24.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.236+0000 7f244dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.236+0000 7f244dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.236+0000 7f244dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.237+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.237+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.237+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-07-27T14:41:24.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 -- 172.21.15.87:0/3637172810 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 msgr2=0x7f245013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 --2- 172.21.15.87:0/3637172810 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f245013bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 -- 172.21.15.87:0/3637172810 >> v2:172.21.15.87:3300/0 conn(0x7f245013ed50 msgr2=0x7f245013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 --2- 172.21.15.87:0/3637172810 >> v2:172.21.15.87:3300/0 conn(0x7f245013ed50 0x7f245013f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f244dd9b640 1 -- 172.21.15.87:0/3637172810 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24500ab910 con 0x7f24500aad90 2024-07-27T14:41:24.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.239+0000 7f631ceee640 1 --2- 172.21.15.87:0/630505457 >> v2:172.21.15.112:3300/0 conn(0x7f63180b1a80 0x7f631813bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.242+0000 7ff9d77fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff9d80ad460 0x7ff9d80737f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36762/0 (socket says 172.21.15.87:36762) 2024-07-27T14:41:24.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.242+0000 7ff9d77fe640 1 -- 172.21.15.87:0/3150861961 learned_addr learned my addr 172.21.15.87:0/3150861961 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.243+0000 7f631ceee640 10 monclient(hunting): get_auth_request con 0x7f63180b1a80 auth_method 0 2024-07-27T14:41:24.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.243+0000 7f631ceee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.243+0000 7f631ceee640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.243+0000 7f631ceee640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.244+0000 7f631ceee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.244+0000 7f631ceee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.244+0000 7f631ceee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.244+0000 7f1e26d9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1e280b1c80 0x7f1e2813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36776/0 (socket says 172.21.15.87:36776) 2024-07-27T14:41:24.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.244+0000 7f1e26d9d640 1 -- 172.21.15.87:0/3532748080 learned_addr learned my addr 172.21.15.87:0/3532748080 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.246+0000 7f244dd9b640 1 --2- 172.21.15.87:0/3637172810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f245013c2b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f24500acd30 tx=0x7f244002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34ee91c961bac71 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.249+0000 7faa78b9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7faa740b4a50 0x7faa74146010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36788/0 (socket says 172.21.15.87:36788) 2024-07-27T14:41:24.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.249+0000 7faa78b9d640 1 -- 172.21.15.87:0/2159569748 learned_addr learned my addr 172.21.15.87:0/2159569748 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.250+0000 7f631ceee640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 1 -- 172.21.15.87:0/630505457 >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 msgr2=0x7f63180ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 1 --2- 172.21.15.87:0/630505457 >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 0x7f63180ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 1 -- 172.21.15.87:0/630505457 >> v1:172.21.15.112:6789/0 conn(0x7f63180ac340 legacy=0x7f63180addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.251+0000 7f631ceee640 1 -- 172.21.15.87:0/630505457 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63180ab980 con 0x7f63180b1a80 2024-07-27T14:41:24.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.254+0000 7f631ceee640 1 --2- 172.21.15.87:0/630505457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63180b1a80 0x7f631813bde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63180ad600 tx=0x7f630c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7607278a474f0ed9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 1 -- 172.21.15.87:0/3637172810 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440038070 con 0x7f24500aad90 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: dump: 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.258 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244f59e640 5 monclient: authenticate success, global_id 4306 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 1 -- 172.21.15.87:0/3637172810 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2440031700 con 0x7f24500aad90 2024-07-27T14:41:24.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244d59a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244f59e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.257+0000 7f244f59e640 4 set_mon_vals no callback set 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244d59a640 1 -- 172.21.15.87:0/3637172810 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440031ad0 con 0x7f24500aad90 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244d59a640 10 monclient: dump: 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.260 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.261 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.261 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 10 monclient: shutdown 2024-07-27T14:41:24.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 -- 172.21.15.87:0/3637172810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 msgr2=0x7f245013c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 --2- 172.21.15.87:0/3637172810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f245013c2b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f24500acd30 tx=0x7f244002b7a0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 -- 172.21.15.87:0/3637172810 shutdown_connections 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 --2- 172.21.15.87:0/3637172810 >> v2:172.21.15.87:3300/0 conn(0x7f245013ed50 0x7f245013f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 --2- 172.21.15.87:0/3637172810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f245013c2b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 --2- 172.21.15.87:0/3637172810 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f245013bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 -- 172.21.15.87:0/3637172810 >> 172.21.15.87:0/3637172810 conn(0x7f2450067e10 msgr2=0x7f24500b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 -- 172.21.15.87:0/3637172810 shutdown_connections 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.258+0000 7f244f59e640 1 -- 172.21.15.87:0/3637172810 wait complete. 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.259+0000 7f244f59e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.259+0000 7f244f59e640 10 monclient: monmap: 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.263 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 librados: starting msgr at 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 librados: starting objecter 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 Processor -- start 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 -- start start 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 librados: setting wanted keys 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 1 librados: calling monclient init 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.260+0000 7f244f59e640 10 monclient: init 2024-07-27T14:41:24.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f24501d3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f24500aa9b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f24501d4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f24500aad90 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f245013ed50 0x7f24501d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f245013ed50 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f245013f720 con 0x7f245013ed50 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f24500b4eb0 con 0x7f24500aad90 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f245013f8a0 con 0x7f24500aa9b0 2024-07-27T14:41:24.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.261+0000 7f244f59e640 10 monclient(hunting): authenticate will time out at 1224.287536s 2024-07-27T14:41:24.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.264+0000 7f63177fe640 1 -- 172.21.15.87:0/630505457 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c0315a0 con 0x7f63180b1a80 2024-07-27T14:41:24.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: dump: 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 1 -- 172.21.15.87:0/630505457 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f630c037070 con 0x7f63180b1a80 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 5 monclient: authenticate success, global_id 4309 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 1 -- 172.21.15.87:0/630505457 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c031960 con 0x7f63180b1a80 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 4 set_mon_vals no callback set 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f63177fe640 10 monclient: dump: 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7ff9d7fff640 1 --2- 172.21.15.87:0/3150861961 >> v2:172.21.15.87:3300/0 conn(0x7ff9d80b1b80 0x7ff9d80aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 10 monclient: shutdown 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 1 -- 172.21.15.87:0/630505457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63180b1a80 msgr2=0x7f631813bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f631def0640 1 --2- 172.21.15.87:0/630505457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63180b1a80 0x7f631813bde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63180ad600 tx=0x7f630c02b840 comp rx=0 tx=0).stop 2024-07-27T14:41:24.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.265+0000 7f244dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f24501d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 -- 172.21.15.87:0/630505457 shutdown_connections 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 --2- 172.21.15.87:0/630505457 >> v2:172.21.15.87:3300/0 conn(0x7f631813c320 0x7f63180ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 --2- 172.21.15.87:0/630505457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63180b1a80 0x7f631813bde0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 -- 172.21.15.87:0/630505457 >> 172.21.15.87:0/630505457 conn(0x7f63180b4c30 msgr2=0x7f63180b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 -- 172.21.15.87:0/630505457 shutdown_connections 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f3f8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f3f8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f3f8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f631def0640 1 -- 172.21.15.87:0/630505457 wait complete. 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f24501d4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58436/0 (socket says 172.21.15.87:58436) 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 1 -- 172.21.15.87:0/860260179 learned_addr learned my addr 172.21.15.87:0/860260179 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 10 monclient(hunting): get_auth_request con 0x7f24500aad90 auth_method 0 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.266+0000 7f244dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 10 monclient: monmap: 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.273 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 librados: starting msgr at 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 librados: starting objecter 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f244dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 Processor -- start 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 -- start start 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 librados: setting wanted keys 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 1 librados: calling monclient init 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.267+0000 7f631def0640 10 monclient: init 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:24.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63180ac340 0x7f631806e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): picked mon.noname-c con 0x7f63180ac340 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63180b1a80 0x7f631813fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): picked mon.noname-a con 0x7f63180b1a80 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 0x7f6318140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): picked mon.noname-b con 0x7f631813c320 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63180ab210 con 0x7f63180b1a80 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f631813c730 con 0x7f631813c320 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f63180ab4f0 con 0x7f63180ac340 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.268+0000 7f631def0640 10 monclient(hunting): authenticate will time out at 1224.294709s 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 -- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 msgr2=0x7f24501d3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24500aa9b0 0x7f24501d3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 -- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f245013ed50 msgr2=0x7f24501d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f245013ed50 0x7f24501d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.270+0000 7f244dd9b640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245006f3f0 con 0x7f24500aad90 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 1 -- 172.21.15.87:0/2786497930 >> v1:172.21.15.112:6789/0 conn(0x7f3f840ab2e0 legacy=0x7f3f840b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 1 -- 172.21.15.87:0/2786497930 >> v1:172.21.15.87:6789/0 conn(0x7f3f840b0720 legacy=0x7f3f8415b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f3f8259c640 1 -- 172.21.15.87:0/2786497930 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f840ad540 con 0x7f3f840b19f0 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9d80b1b80 auth_method 0 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.272+0000 7f631d6ef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 0x7f6318140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f1e2659c640 1 --2- 172.21.15.87:0/3532748080 >> v2:172.21.15.87:3300/0 conn(0x7f1e28072b70 0x7f1e280b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 1 -- 172.21.15.87:0/4209712603 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0039070 con 0x7f9ba80734a0 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: dump: 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.282 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bb041a640 5 monclient: authenticate success, global_id 4265 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bb041a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 1 -- 172.21.15.87:0/4209712603 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba002cc60 con 0x7f9ba80734a0 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bb041a640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bb041a640 4 set_mon_vals no callback set 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 1 -- 172.21.15.87:0/4209712603 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0034660 con 0x7f9ba80734a0 2024-07-27T14:41:24.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.273+0000 7f9bad1b4640 10 monclient: dump: 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.284 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 0x7f6318140350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58450/0 (socket says 172.21.15.87:58450) 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f1e2659c640 10 monclient(hunting): get_auth_request con 0x7f1e28072b70 auth_method 0 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f1e2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f1e2659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f1e2659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 1 -- 172.21.15.87:0/3062651952 learned_addr learned my addr 172.21.15.87:0/3062651952 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): get_auth_request con 0x7f631813c320 auth_method 0 2024-07-27T14:41:24.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 10 monclient: shutdown 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 -- 172.21.15.87:0/4209712603 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 msgr2=0x7f9ba81442d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 --2- 172.21.15.87:0/4209712603 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 0x7f9ba81442d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002beb0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f631d6ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 -- 172.21.15.87:0/4209712603 shutdown_connections 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 --2- 172.21.15.87:0/4209712603 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80734a0 0x7f9ba81442d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 --2- 172.21.15.87:0/4209712603 >> v2:172.21.15.112:3300/0 conn(0x7f9ba80ac200 0x7f9ba80ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 -- 172.21.15.87:0/4209712603 >> 172.21.15.87:0/4209712603 conn(0x7f9ba80b49f0 msgr2=0x7f9ba80b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 -- 172.21.15.87:0/4209712603 shutdown_connections 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.274+0000 7f9bb041a640 1 -- 172.21.15.87:0/4209712603 wait complete. 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 10 monclient: monmap: 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 librados: starting msgr at 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 librados: starting objecter 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 Processor -- start 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 -- start start 2024-07-27T14:41:24.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 librados: setting wanted keys 2024-07-27T14:41:24.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 1 librados: calling monclient init 2024-07-27T14:41:24.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.275+0000 7f9bb041a640 10 monclient: init 2024-07-27T14:41:24.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-07-27T14:41:24.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 1 -- 172.21.15.87:0/3150861961 >> v1:172.21.15.112:6789/0 conn(0x7ff9d80ad460 legacy=0x7ff9d80737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 1 -- 172.21.15.87:0/3150861961 >> v1:172.21.15.87:6789/0 conn(0x7ff9d8142cb0 legacy=0x7ff9d814a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 1 monclient: found mon.noname-a 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7ff9d7fff640 1 -- 172.21.15.87:0/3150861961 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d80abf00 con 0x7ff9d80b1b80 2024-07-27T14:41:24.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 0x7f9ba80b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba80734a0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80abe20 0x7f9ba81452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba80abe20 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ba80ac200 0x7f9ba8138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba80ac200 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8151b50 con 0x7f9ba80ac200 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81519d0 con 0x7f9ba80734a0 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba8151cd0 con 0x7f9ba80abe20 2024-07-27T14:41:24.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f9bb041a640 10 monclient(hunting): authenticate will time out at 1224.302632s 2024-07-27T14:41:24.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.276+0000 7f3f8259c640 1 --2- 172.21.15.87:0/2786497930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 0x7f3f840aada0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097d0 tx=0x7f3f7402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c978bf4b86ce4d79 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f1e2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f1e2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f1e2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 -- 172.21.15.87:0/3062651952 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63180ac340 msgr2=0x7f631806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63180ac340 0x7f631806e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 -- 172.21.15.87:0/3062651952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63180b1a80 msgr2=0x7f631813fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63180b1a80 0x7f631813fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f631d6ef640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631806fa70 con 0x7f631813c320 2024-07-27T14:41:24.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.277+0000 7f27baffd640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f27b402be60 0x7f27b402f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 1 -- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 msgr2=0x7f1e280b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 1 --2- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e280b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 1 -- 172.21.15.87:0/3532748080 >> v1:172.21.15.112:6789/0 conn(0x7f1e280b1c80 legacy=0x7f1e2813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 1 monclient: found mon.noname-a 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.279+0000 7f1e2659c640 1 -- 172.21.15.87:0/3532748080 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e281404b0 con 0x7f1e28072b70 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 1 -- 172.21.15.87:0/2786497930 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74031780 con 0x7f3f840b19f0 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 10 monclient: dump: 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.297 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8359e640 5 monclient: authenticate success, global_id 4271 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f3f8359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f631d6ef640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 0x7f6318140350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6310007ad0 tx=0x7f6310007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f63157fa640 10 client.?.objecter ms_handle_connect 0x7f631813c320 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7f63157fa640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.280+0000 7ff9d7fff640 1 --2- 172.21.15.87:0/3150861961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80b1b80 0x7ff9d80aeac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dc6366b094f2c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 0x7f9ba80b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39172/0 (socket says 172.21.15.87:39172) 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 1 -- 172.21.15.87:0/1560567363 learned_addr learned my addr 172.21.15.87:0/1560567363 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request con 0x7f27b4006bf0 auth_method 0 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f27bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 0x7f9ba80b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58458/0 (socket says 172.21.15.87:58458) 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 1 -- 172.21.15.87:0/3063704240 learned_addr learned my addr 172.21.15.87:0/3063704240 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 10 monclient(hunting): get_auth_request con 0x7f9ba80734a0 auth_method 0 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.281+0000 7f9bae1b6640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f9bae1b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f9bae1b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f9bae1b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f27baffd640 10 monclient(hunting): get_auth_request con 0x7f27b402be60 auth_method 0 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f27baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f27baffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.282+0000 7f27baffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f244dd9b640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f24501d4000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2440009890 tx=0x7f24400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f243effd640 10 client.?.objecter ms_handle_connect 0x7f24500aad90 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f243effd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.283+0000 7f1e2659c640 1 --2- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 0x7f1e280b4ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e180097a0 tx=0x7f1e1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fd8016e9d5406e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.284+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.284+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.284+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7faa79b9f640 1 --2- 172.21.15.87:0/2159569748 >> v2:172.21.15.87:3300/0 conn(0x7faa740aa890 0x7faa7414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 1 -- 172.21.15.87:0/1560567363 >> v2:172.21.15.87:3300/0 conn(0x7f27b402be60 msgr2=0x7f27b402f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 1 --2- 172.21.15.87:0/1560567363 >> v2:172.21.15.87:3300/0 conn(0x7f27b402be60 0x7f27b402f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 1 -- 172.21.15.87:0/1560567363 >> v1:172.21.15.87:6789/0 conn(0x7f27b40315a0 legacy=0x7f27b40cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.285+0000 7f27bb7fe640 1 -- 172.21.15.87:0/1560567363 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b402c340 con 0x7f27b4006bf0 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 -- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80abe20 msgr2=0x7f9ba81452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9ba80abe20 0x7f9ba81452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 -- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ba80ac200 msgr2=0x7f9ba8138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9ba80ac200 0x7f9ba8138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.286+0000 7f9bae1b6640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba813a000 con 0x7f9ba80734a0 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 1 -- 172.21.15.87:0/2786497930 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f74037070 con 0x7f3f840b19f0 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 1 -- 172.21.15.87:0/2786497930 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74031b40 con 0x7f3f840b19f0 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8359e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8359e640 4 set_mon_vals no callback set 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.287+0000 7f3f8159a640 10 monclient: dump: 2024-07-27T14:41:24.309 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 10 monclient: shutdown 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 -- 172.21.15.87:0/2786497930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 msgr2=0x7f3f840aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 --2- 172.21.15.87:0/2786497930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 0x7f3f840aada0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097d0 tx=0x7f3f7402b820 comp rx=0 tx=0).stop 2024-07-27T14:41:24.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 -- 172.21.15.87:0/2786497930 shutdown_connections 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 --2- 172.21.15.87:0/2786497930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b19f0 0x7f3f840aada0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 -- 172.21.15.87:0/2786497930 >> 172.21.15.87:0/2786497930 conn(0x7f3f840b4c30 msgr2=0x7f3f840b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 -- 172.21.15.87:0/2786497930 shutdown_connections 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.288+0000 7f3f8359e640 1 -- 172.21.15.87:0/2786497930 wait complete. 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.289+0000 7f3f8359e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.289+0000 7f3f8359e640 10 monclient: monmap: 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.311 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.289+0000 7f3f8359e640 1 librados: starting msgr at 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.289+0000 7f3f8359e640 1 librados: starting objecter 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.289+0000 7f3f8359e640 1 Processor -- start 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 1 -- start start 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 1 librados: setting wanted keys 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 1 librados: calling monclient init 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 10 monclient: init 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f840ab2e0 0x7f3f8413fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840ab2e0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.290+0000 7f3f8359e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 0x7f3f84072030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f840b0720 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f840b19f0 0x7f3f841e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f840b19f0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8408cc20 con 0x7f3f840b19f0 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3f840aebb0 con 0x7f3f840ab2e0 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8408cda0 con 0x7f3f840b0720 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f9bae1b6640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 0x7f9ba80b3b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c4d0 tx=0x7f9b9800c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f9b9effd640 10 client.?.objecter ms_handle_connect 0x7f9ba80734a0 2024-07-27T14:41:24.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f9b9effd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.291+0000 7f3f8359e640 10 monclient(hunting): authenticate will time out at 1224.317094s 2024-07-27T14:41:24.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.293+0000 7f3f81d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 0x7f3f84072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.293+0000 7faa79b9f640 10 monclient(hunting): get_auth_request con 0x7faa740aa890 auth_method 0 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.293+0000 7faa79b9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.293+0000 7faa79b9f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.293+0000 7faa79b9f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.294+0000 7f27bb7fe640 1 --2- 172.21.15.87:0/1560567363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009870 tx=0x7f27b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b8b9c8767c059 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.295+0000 7faa79b9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.295+0000 7faa79b9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.295+0000 7faa79b9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 0x7f3f84072030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39202/0 (socket says 172.21.15.87:39202) 2024-07-27T14:41:24.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 1 -- 172.21.15.87:0/1198674224 learned_addr learned my addr 172.21.15.87:0/1198674224 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 10 monclient(hunting): get_auth_request con 0x7f3f840b0720 auth_method 0 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.296+0000 7f3f81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7f3f81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7f3f81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7faa79b9f640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-07-27T14:41:24.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7f3f81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7faa79b9f640 1 -- 172.21.15.87:0/2159569748 >> v1:172.21.15.112:6789/0 conn(0x7faa740b4a50 legacy=0x7faa74146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.297+0000 7faa79b9f640 1 -- 172.21.15.87:0/2159569748 >> v1:172.21.15.87:6789/0 conn(0x7faa740728e0 legacy=0x7faa7413fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.298+0000 7faa79b9f640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.298+0000 7faa79b9f640 1 monclient: found mon.noname-a 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.298+0000 7faa79b9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.298+0000 7faa79b9f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.298+0000 7faa79b9f640 1 -- 172.21.15.87:0/2159569748 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa740b2cb0 con 0x7faa740aa890 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.300+0000 7f3f81d9b640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.300+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310018070 con 0x7f631813c320 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 -- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f840ab2e0 msgr2=0x7f3f8413fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f840ab2e0 0x7f3f8413fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 -- 172.21.15.87:0/1198674224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f840b19f0 msgr2=0x7f3f841e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: dump: 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.320 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f840b19f0 0x7f3f841e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f3f81d9b640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f84141870 con 0x7f3f840b0720 2024-07-27T14:41:24.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6310003da0 con 0x7f631813c320 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 5 monclient: authenticate success, global_id 4315 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 10 monclient: _renew_subs 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631806fd50 con 0x7f631813c320 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631000dab0 con 0x7f631813c320 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 10 monclient: _renew_subs 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f6316ffd640 4 set_mon_vals no callback set 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63181427d0 con 0x7f631813c320 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f631def0640 1 librados: init done 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7f63157fa640 10 monclient: dump: 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.326 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.301+0000 7faa79b9f640 1 --2- 172.21.15.87:0/2159569748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740aa890 0x7faa7414e390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009d90 tx=0x7faa6c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab42d75ab5945a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 15 client.4315.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 20 client.4315.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6318142e40 con 0x7f631813c320 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440043020 con 0x7f24500aad90 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: dump: 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.329 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2440033920 con 0x7f24500aad90 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440038040 con 0x7f24500aad90 2024-07-27T14:41:24.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243effd640 10 monclient: dump: 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.302+0000 7f243ffff640 4 set_mon_vals no callback set 2024-07-27T14:41:24.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 5 monclient: authenticate success, global_id 4312 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 10 monclient: _renew_subs 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245006f660 con 0x7f24500aad90 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 10 monclient: _renew_subs 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24501d4540 con 0x7f24500aad90 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.303+0000 7f244f59e640 1 librados: init done 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 15 client.4312.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 20 client.4312.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.304+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2450142b70 con 0x7f24500aad90 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98017070 con 0x7f9ba80734a0 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 10 monclient: dump: 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.333 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9bb041a640 5 monclient: authenticate success, global_id 4318 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9bb041a640 10 monclient: _renew_subs 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9bb041a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.307+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba81457f0 con 0x7f9ba80734a0 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b98005ce0 con 0x7f9ba80734a0 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98007270 con 0x7f9ba80734a0 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9b9effd640 10 monclient: dump: 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 monclient: _renew_subs 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bac9b3640 4 set_mon_vals no callback set 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81385b0 con 0x7f9ba80734a0 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 1 librados: init done 2024-07-27T14:41:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f3f81d9b640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 0x7f3f84072030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800c3f0 tx=0x7f3f7800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f3f72ffd640 10 client.?.objecter ms_handle_connect 0x7f3f840b0720 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f3f72ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 15 client.4318.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.308+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ba81442d0 con 0x7f9ba80734a0 2024-07-27T14:41:24.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 1 -- 172.21.15.87:0/3532748080 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180315a0 con 0x7f1e28072b70 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: dump: 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.341 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 1 -- 172.21.15.87:0/3532748080 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e18037070 con 0x7f1e28072b70 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 1 -- 172.21.15.87:0/3532748080 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e18031960 con 0x7f1e28072b70 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2ca0e640 5 monclient: authenticate success, global_id 4296 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2ca0e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2ca0e640 4 set_mon_vals no callback set 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.310+0000 7f1e2559a640 10 monclient: dump: 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.342 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 10 monclient: shutdown 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 msgr2=0x7f1e280b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 0x7f1e280b4ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e180097a0 tx=0x7f1e1802b840 comp rx=0 tx=0).stop 2024-07-27T14:41:24.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/3532748080 shutdown_connections 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e280b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/3532748080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 0x7f1e280b4ca0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/3532748080 >> 172.21.15.87:0/3532748080 conn(0x7f1e28068150 msgr2=0x7f1e280b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/3532748080 shutdown_connections 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.311+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/3532748080 wait complete. 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.312+0000 7f1e2ca0e640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.312+0000 7f1e2ca0e640 10 monclient: monmap: 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.344 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.312+0000 7f1e2ca0e640 1 librados: starting msgr at 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.312+0000 7f1e2ca0e640 1 librados: starting objecter 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 Processor -- start 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 -- start start 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 librados: setting wanted keys 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 librados: calling monclient init 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient: init 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 0x7f1e28148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-a con 0x7f1e28072b70 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e281d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-c con 0x7f1e280af970 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e280b1c80 0x7f1e281d9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): picked mon.noname-b con 0x7f1e280b1c80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1e280b2520 con 0x7f1e28072b70 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e280b23a0 con 0x7f1e280b1c80 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2814d510 con 0x7f1e280af970 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.313+0000 7f1e2ca0e640 10 monclient(hunting): authenticate will time out at 1224.339766s 2024-07-27T14:41:24.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 1 -- 172.21.15.87:0/1560567363 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b0035020 con 0x7f27b4006bf0 2024-07-27T14:41:24.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: dump: 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 1 -- 172.21.15.87:0/1560567363 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27b0031640 con 0x7f27b4006bf0 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27c0c4d640 5 monclient: authenticate success, global_id 4274 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27c0c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 1 -- 172.21.15.87:0/1560567363 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b002d040 con 0x7f27b4006bf0 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27c0c4d640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27c0c4d640 4 set_mon_vals no callback set 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.314+0000 7f27ba7fc640 10 monclient: dump: 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.349 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 10 monclient: shutdown 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 -- 172.21.15.87:0/1560567363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 msgr2=0x7f27b4030820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/1560567363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009870 tx=0x7f27b002b8f0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 -- 172.21.15.87:0/1560567363 shutdown_connections 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/1560567363 >> v2:172.21.15.87:3300/0 conn(0x7f27b402be60 0x7f27b402f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/1560567363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b4006bf0 0x7f27b4030820 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 -- 172.21.15.87:0/1560567363 >> 172.21.15.87:0/1560567363 conn(0x7f27b4005a40 msgr2=0x7f27b4005e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 -- 172.21.15.87:0/1560567363 shutdown_connections 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.315+0000 7f27c0c4d640 1 -- 172.21.15.87:0/1560567363 wait complete. 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e281d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f27c0c4d640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f27c0c4d640 10 monclient: monmap: 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.351 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.352 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 1 -- 172.21.15.87:0/3150861961 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0039070 con 0x7ff9d80b1b80 2024-07-27T14:41:24.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:24.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: dump: 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9de862640 5 monclient: authenticate success, global_id 4293 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9de862640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 1 -- 172.21.15.87:0/3150861961 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d002cc60 con 0x7ff9d80b1b80 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9d6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e281d8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39208/0 (socket says 172.21.15.87:39208) 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 1 -- 172.21.15.87:0/1139173911 learned_addr learned my addr 172.21.15.87:0/1139173911 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 10 monclient(hunting): get_auth_request con 0x7f1e280af970 auth_method 0 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f1e25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f27c0c4d640 1 librados: starting msgr at 2024-07-27T14:41:24.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f27c0c4d640 1 librados: starting objecter 2024-07-27T14:41:24.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7f27c0c4d640 1 Processor -- start 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9de862640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.316+0000 7ff9de862640 4 set_mon_vals no callback set 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9d6ffd640 1 -- 172.21.15.87:0/3150861961 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0034660 con 0x7ff9d80b1b80 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9d6ffd640 10 monclient: dump: 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.357 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 10 monclient: shutdown 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 -- 172.21.15.87:0/3150861961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80b1b80 msgr2=0x7ff9d80aeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 -- start start 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 librados: setting wanted keys 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 librados: calling monclient init 2024-07-27T14:41:24.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient: init 2024-07-27T14:41:24.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 --2- 172.21.15.87:0/3150861961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80b1b80 0x7ff9d80aeac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002beb0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 -- 172.21.15.87:0/3150861961 shutdown_connections 2024-07-27T14:41:24.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 --2- 172.21.15.87:0/3150861961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80b1b80 0x7ff9d80aeac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 -- 172.21.15.87:0/3150861961 >> 172.21.15.87:0/3150861961 conn(0x7ff9d80b4d30 msgr2=0x7ff9d80b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 -- 172.21.15.87:0/3150861961 shutdown_connections 2024-07-27T14:41:24.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7ff9de862640 1 -- 172.21.15.87:0/3150861961 wait complete. 2024-07-27T14:41:24.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27b4006bf0 0x7f27b4029ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f27b4006bf0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 0x7f27b4161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f27b402be60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b40315a0 0x7f27b4161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f27b40315a0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4031a50 con 0x7f27b4006bf0 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4034680 con 0x7f27b40315a0 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f27b4031bd0 con 0x7f27b402be60 2024-07-27T14:41:24.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.317+0000 7f27c0c4d640 10 monclient(hunting): authenticate will time out at 1224.343934s 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 10 monclient: monmap: 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.362 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 librados: starting msgr at 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 librados: starting objecter 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 Processor -- start 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 -- start start 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 librados: setting wanted keys 2024-07-27T14:41:24.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 1 librados: calling monclient init 2024-07-27T14:41:24.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.318+0000 7ff9de862640 10 monclient: init 2024-07-27T14:41:24.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7f1e25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7f1e25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7f1e25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80ad460 0x7ff9d8138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d80ad460 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 0x7ff9d8139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d80b1b80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9d8142cb0 0x7ff9d8135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d8142cb0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80ac900 con 0x7ff9d80ad460 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80ac780 con 0x7ff9d8142cb0 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d8143100 con 0x7ff9d80b1b80 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.319+0000 7ff9de862640 10 monclient(hunting): authenticate will time out at 1224.345303s 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f780052d0 con 0x7f3f840b0720 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: dump: 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.367 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 5 monclient: authenticate success, global_id 4277 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 10 monclient: _renew_subs 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8413f4f0 con 0x7f3f840b0720 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f78005470 con 0x7f3f840b0720 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f780052d0 con 0x7f3f840b0720 2024-07-27T14:41:24.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f72ffd640 10 monclient: dump: 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f80d99640 4 set_mon_vals no callback set 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 10 monclient: _renew_subs 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f84140b00 con 0x7f3f840b0720 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.320+0000 7f3f8359e640 1 librados: init done 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 15 client.4277.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.321+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f84069180 con 0x7f3f840b0720 2024-07-27T14:41:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-07-27T14:41:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 -- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e280b1c80 msgr2=0x7f1e281d9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e280b1c80 0x7f1e281d9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 -- 172.21.15.87:0/1139173911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 msgr2=0x7f1e28148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1e28072b70 0x7f1e28148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.323+0000 7f1e25d9b640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2814a720 con 0x7f1e280af970 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 1 -- 172.21.15.87:0/2159569748 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c039070 con 0x7faa740aa890 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: dump: 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.373 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 1 -- 172.21.15.87:0/2159569748 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c02cb40 con 0x7faa740aa890 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 1 -- 172.21.15.87:0/2159569748 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c034440 con 0x7faa740aa890 2024-07-27T14:41:24.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa67fff640 10 monclient: dump: 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa7a3a0640 5 monclient: authenticate success, global_id 4302 2024-07-27T14:41:24.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa7a3a0640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.324+0000 7faa7a3a0640 4 set_mon_vals no callback set 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 10 monclient: shutdown 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2159569748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740aa890 msgr2=0x7faa7414e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 --2- 172.21.15.87:0/2159569748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740aa890 0x7faa7414e390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009d90 tx=0x7faa6c02bd90 comp rx=0 tx=0).stop 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2159569748 shutdown_connections 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 --2- 172.21.15.87:0/2159569748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740aa890 0x7faa7414e390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2159569748 >> 172.21.15.87:0/2159569748 conn(0x7faa740682b0 msgr2=0x7faa740b8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2159569748 shutdown_connections 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.325+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2159569748 wait complete. 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 10 monclient: monmap: 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.376 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.377 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7f1e25d9b640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e281d8b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b450 tx=0x7f1e1c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7f1e16ffd640 10 client.?.objecter ms_handle_connect 0x7f1e280af970 2024-07-27T14:41:24.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7f1e16ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 librados: starting msgr at 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 librados: starting objecter 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 Processor -- start 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 -- start start 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 librados: setting wanted keys 2024-07-27T14:41:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 1 librados: calling monclient init 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.326+0000 7faa7a3a0640 10 monclient: init 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 0x7faa7413c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-b con 0x7faa740728e0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa740aa890 0x7faa741dbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-c con 0x7faa740aa890 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740b4a50 0x7faa741dc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): picked mon.noname-a con 0x7faa740b4a50 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7faa74140550 con 0x7faa740b4a50 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faa740b42e0 con 0x7faa740728e0 2024-07-27T14:41:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7faa7414e8d0 con 0x7faa740aa890 2024-07-27T14:41:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.327+0000 7faa7a3a0640 10 monclient(hunting): authenticate will time out at 1224.353842s 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.336+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2440046020 con 0x7f24500aad90 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.337+0000 7f243effd640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f243006d840 0x7f243006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.337+0000 7f244e59c640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f243006d840 0x7f243006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.337+0000 7f244e59c640 10 monclient: get_auth_request con 0x7f243006d840 auth_method 0 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.338+0000 7f243effd640 10 client.4312.objecter ms_handle_connect 0x7f243006d840 2024-07-27T14:41:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.338+0000 7f244e59c640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f243006d840 0x7f243006fd00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2448007250 tx=0x7f244800f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f2440079270 con 0x7f24500aad90 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 10 client.4312.objecter ms_dispatch 0x7f2450000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 3 client.4312.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 3 client.4312.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 10 client.4312.objecter checking command 1 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f2430073c80 0x7f24300760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter _get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 15 client.4312.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter _get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 15 client.4312.objecter _session_command_op_assign 2 1 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 4 2024-07-27T14:41:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 10 client.4312.objecter _send_command 1 2024-07-27T14:41:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2440037e90 con 0x7f2430073c80 2024-07-27T14:41:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f243effd640 20 client.4312.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.346+0000 7f244ed9d640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f2430073c80 0x7f24300760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.347+0000 7f244ed9d640 10 monclient: get_auth_request con 0x7f2430073c80 auth_method 0 2024-07-27T14:41:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.348+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3f780067d0 con 0x7f3f840b0720 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.349+0000 7f3f72ffd640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f6406dbc0 0x7f3f64070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.349+0000 7f3f8259c640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f6406dbc0 0x7f3f64070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.349+0000 7f3f8259c640 10 monclient: get_auth_request con 0x7f3f6406dbc0 auth_method 0 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.350+0000 7f3f72ffd640 10 client.4277.objecter ms_handle_connect 0x7f3f6406dbc0 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.350+0000 7f3f8259c640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f6406dbc0 0x7f3f64070080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f74036000 tx=0x7f3f74030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.351+0000 7f243effd640 10 client.4312.objecter ms_handle_connect 0x7f2430073c80 2024-07-27T14:41:24.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.351+0000 7f244ed9d640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f2430073c80 0x7f24300760c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.358+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6310005d10 con 0x7f631813c320 2024-07-27T14:41:24.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f63157fa640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f62fc06d810 0x7f62fc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f631ceee640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f62fc06d810 0x7f62fc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f631ceee640 10 monclient: get_auth_request con 0x7f62fc06d810 auth_method 0 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f3f780050d0 con 0x7f3f840b0720 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f3f72ffd640 10 client.4277.objecter ms_dispatch 0x7f3f84000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f3f72ffd640 3 client.4277.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f3f72ffd640 3 client.4277.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.359+0000 7f3f72ffd640 10 client.4277.objecter checking command 1 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f3f64074000 0x7f3f64076440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter _get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 15 client.4277.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter _get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 15 client.4277.objecter _session_command_op_assign 5 1 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 4 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 10 client.4277.objecter _send_command 1 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f7800ee50 con 0x7f3f64074000 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f72ffd640 20 client.4277.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 10 client.4315.objecter ms_handle_connect 0x7f62fc06d810 2024-07-27T14:41:24.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f631ceee640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f62fc06d810 0x7f62fc06fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f630c036000 tx=0x7f630c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f27baffd640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 0x7f27b4161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f3f82d9d640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f3f64074000 0x7f3f64076440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f6310057210 con 0x7f631813c320 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 10 client.4315.objecter ms_dispatch 0x7f6318000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 3 client.4315.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 3 client.4315.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 10 client.4315.objecter checking command 1 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.360+0000 7f63157fa640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f62fc073c50 0x7f62fc076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter _get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 15 client.4315.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter _get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 15 client.4315.objecter _session_command_op_assign 6 1 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 4 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 10 client.4315.objecter _send_command 1 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6310017ec0 con 0x7f62fc073c50 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f63157fa640 20 client.4315.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f3f82d9d640 10 monclient: get_auth_request con 0x7f3f64074000 auth_method 0 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 0x7f27b4161210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39216/0 (socket says 172.21.15.87:39216) 2024-07-27T14:41:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 1 -- 172.21.15.87:0/3843126661 learned_addr learned my addr 172.21.15.87:0/3843126661 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 10 monclient(hunting): get_auth_request con 0x7f27b402be60 auth_method 0 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f27baffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 0x7ff9d8139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.361+0000 7f6317fff640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f62fc073c50 0x7f62fc076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7f27baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7f6317fff640 10 monclient: get_auth_request con 0x7f62fc073c50 auth_method 0 2024-07-27T14:41:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 0x7ff9d8139480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39222/0 (socket says 172.21.15.87:39222) 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 1 -- 172.21.15.87:0/4223377180 learned_addr learned my addr 172.21.15.87:0/4223377180 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request con 0x7ff9d80b1b80 auth_method 0 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.362+0000 7ff9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.363+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.363+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.363+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.364+0000 7f3f82d9d640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f3f64074000 0x7f3f64076440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.364+0000 7f3f72ffd640 10 client.4277.objecter ms_handle_connect 0x7f3f64074000 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 -- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b40315a0 msgr2=0x7f27b4161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b40315a0 0x7f27b4161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 -- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27b4006bf0 msgr2=0x7f27b4029ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27b4006bf0 0x7f27b4029ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f27baffd640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b402b8f0 con 0x7f27b402be60 2024-07-27T14:41:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c015020 con 0x7f1e280af970 2024-07-27T14:41:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: dump: 2024-07-27T14:41:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c011890 con 0x7f1e280af970 2024-07-27T14:41:24.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c011c80 con 0x7f1e280af970 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e16ffd640 10 monclient: dump: 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.400 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e24d99640 4 set_mon_vals no callback set 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 5 monclient: authenticate success, global_id 4280 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 10 monclient: _renew_subs 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2814a990 con 0x7f1e280af970 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 10 monclient: _renew_subs 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.366+0000 7f1e2ca0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e281484a0 con 0x7f1e280af970 2024-07-27T14:41:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 1 librados: init done 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f6317fff640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f62fc073c50 0x7f62fc076090 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f63157fa640 10 client.4315.objecter ms_handle_connect 0x7f62fc073c50 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 15 client.4280.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.367+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e280b4ca0 con 0x7f1e280af970 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 -- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9d8142cb0 msgr2=0x7ff9d8135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9d8142cb0 0x7ff9d8135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 -- 172.21.15.87:0/4223377180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80ad460 msgr2=0x7ff9d8138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff9d80ad460 0x7ff9d8138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.370+0000 7ff9d77fe640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d81384d0 con 0x7ff9d80b1b80 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.374+0000 7f27baffd640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 0x7f27b4161210 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f27a800e870 tx=0x7f27a800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.374+0000 7f27a7fff640 10 client.?.objecter ms_handle_connect 0x7f27b402be60 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.374+0000 7f27a7fff640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.374+0000 7ff9d77fe640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 0x7ff9d8139480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c430 tx=0x7ff9cc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.375+0000 7ff9d4ff9640 10 client.?.objecter ms_handle_connect 0x7ff9d80b1b80 2024-07-27T14:41:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.375+0000 7ff9d4ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.380+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9b9801a070 con 0x7f9ba80734a0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.380+0000 7f9b9effd640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9b8c06d7c0 0x7f9b8c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.381+0000 7f9bad9b5640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9b8c06d7c0 0x7f9b8c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.381+0000 7f9bad9b5640 10 monclient: get_auth_request con 0x7f9b8c06d7c0 auth_method 0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.381+0000 7f9b9effd640 10 client.4318.objecter ms_handle_connect 0x7f9b8c06d7c0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.381+0000 7f9bad9b5640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9b8c06d7c0 0x7f9b8c06fc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80b4350 tx=0x7f9ba4006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f9b98056d30 con 0x7f9ba80734a0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 10 client.4318.objecter ms_dispatch 0x7f9ba8000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 3 client.4318.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 3 client.4318.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 10 client.4318.objecter checking command 1 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7f9b8c073c00 0x7f9b8c076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter _get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 15 client.4318.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter _get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 15 client.4318.objecter _session_command_op_assign 1 1 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 4 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 10 client.4318.objecter _send_command 1 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b9800ee50 con 0x7f9b8c073c00 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9b9effd640 20 client.4318.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.384+0000 7f9bae9b7640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7f9b8c073c00 0x7f9b8c076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.385+0000 7f9bae9b7640 10 monclient: get_auth_request con 0x7f9b8c073c00 auth_method 0 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.386+0000 7faa7939e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 0x7faa7413c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 0x7faa7413c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58510/0 (socket says 172.21.15.87:58510) 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 1 -- 172.21.15.87:0/2042148412 learned_addr learned my addr 172.21.15.87:0/2042148412 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): get_auth_request con 0x7faa740728e0 auth_method 0 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.387+0000 7faa7939e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.389+0000 7f9b9effd640 10 client.4318.objecter ms_handle_connect 0x7f9b8c073c00 2024-07-27T14:41:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.389+0000 7f9bae9b7640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7f9b8c073c00 0x7f9b8c076040 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 -- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa740aa890 msgr2=0x7faa741dbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa740aa890 0x7faa741dbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 -- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740b4a50 msgr2=0x7faa741dc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa740b4a50 0x7faa741dc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.390+0000 7faa7939e640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa740acf40 con 0x7faa740728e0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.393+0000 7faa7939e640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 0x7faa7413c690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c410 tx=0x7faa6800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.393+0000 7faa65ffb640 10 client.?.objecter ms_handle_connect 0x7faa740728e0 2024-07-27T14:41:24.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.393+0000 7faa65ffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa68017070 con 0x7faa740728e0 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: dump: 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.412 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa680035f0 con 0x7faa740728e0 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa68003a20 con 0x7faa740728e0 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa65ffb640 10 monclient: dump: 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.413 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa677fe640 4 set_mon_vals no callback set 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 5 monclient: authenticate success, global_id 4324 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 10 monclient: _renew_subs 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa740ad220 con 0x7faa740728e0 2024-07-27T14:41:24.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 10 monclient: _renew_subs 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.398+0000 7faa7a3a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.399+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa740ac230 con 0x7faa740728e0 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.399+0000 7faa7a3a0640 1 librados: init done 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 15 client.4324.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.400+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa740aa460 con 0x7faa740728e0 2024-07-27T14:41:24.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.401+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e1c018020 con 0x7f1e280af970 2024-07-27T14:41:24.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.401+0000 7f1e16ffd640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e0806d840 0x7f1e0806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.402+0000 7f1e2659c640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e0806d840 0x7f1e0806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.402+0000 7f1e2659c640 10 monclient: get_auth_request con 0x7f1e0806d840 auth_method 0 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.402+0000 7f1e16ffd640 10 client.4280.objecter ms_handle_connect 0x7f1e0806d840 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.402+0000 7f1e2659c640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e0806d840 0x7f1e0806fd00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e18030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f1e1c0579a0 con 0x7f1e280af970 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 10 client.4280.objecter ms_dispatch 0x7f1e28000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 3 client.4280.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 3 client.4280.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 10 client.4280.objecter checking command 1 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f1e08073c80 0x7f1e080760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter _get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 15 client.4280.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter _get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 15 client.4280.objecter _session_command_op_assign 4 1 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 4 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 10 client.4280.objecter _send_command 1 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e08076790 con 0x7f1e08073c80 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.406+0000 7f1e16ffd640 20 client.4280.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.407+0000 7f1e26d9d640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f1e08073c80 0x7f1e080760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.407+0000 7f1e26d9d640 10 monclient: get_auth_request con 0x7f1e08073c80 auth_method 0 2024-07-27T14:41:24.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc017070 con 0x7ff9d80b1b80 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: dump: 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9cc005ce0 con 0x7ff9d80b1b80 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0072f0 con 0x7ff9d80b1b80 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d4ff9640 10 monclient: dump: 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.420 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 5 monclient: authenticate success, global_id 4286 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 10 monclient: _renew_subs 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d81387b0 con 0x7ff9d80b1b80 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9d67fc640 4 set_mon_vals no callback set 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 10 monclient: _renew_subs 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d8135d90 con 0x7ff9d80b1b80 2024-07-27T14:41:24.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.410+0000 7ff9de862640 1 librados: init done 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7f1e16ffd640 10 client.4280.objecter ms_handle_connect 0x7f1e08073c80 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7f1e26d9d640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f1e08073c80 0x7f1e080760c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 15 client.4286.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 20 client.4286.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.411+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9d8143490 con 0x7ff9d80b1b80 2024-07-27T14:41:24.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.414+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== osd.5 v2:172.21.15.112:6810/3751677016 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f7800ee50 con 0x7f3f64074000 2024-07-27T14:41:24.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.414+0000 7f3f72ffd640 10 client.4277.objecter ms_dispatch 0x7f3f84000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.414+0000 7f3f72ffd640 10 client.4277.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.414+0000 7f3f72ffd640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 4 2024-07-27T14:41:24.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.414+0000 7f3f72ffd640 15 client.4277.objecter _session_command_op_remove 5 1 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 15 client.4277.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 20 client.4277.objecter _get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 20 client.4277.objecter _get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 15 client.4277.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 20 client.4277.objecter get_session s=0x7f3f64073930 osd=5 3 2024-07-27T14:41:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 15 client.4277.objecter _session_command_op_assign 5 2 2024-07-27T14:41:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 4 2024-07-27T14:41:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 10 client.4277.objecter _send_command 2 2024-07-27T14:41:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.429+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 --> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3f841e93d0 con 0x7f3f64074000 2024-07-27T14:41:24.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== osd.1 v2:172.21.15.87:6800/1509783986 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9800ee50 con 0x7f9b8c073c00 2024-07-27T14:41:24.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f9b9effd640 10 client.4318.objecter ms_dispatch 0x7f9ba8000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f9b9effd640 10 client.4318.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f9b9effd640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 4 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f9b9effd640 15 client.4318.objecter _session_command_op_remove 1 1 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== osd.6 v2:172.21.15.112:6802/591715754 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6310017ec0 con 0x7f62fc073c50 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f63157fa640 10 client.4315.objecter ms_dispatch 0x7f6318000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f63157fa640 10 client.4315.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f63157fa640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 4 2024-07-27T14:41:24.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.436+0000 7f63157fa640 15 client.4315.objecter _session_command_op_remove 6 1 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== osd.2 v2:172.21.15.87:6820/357294290 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2440037e90 con 0x7f2430073c80 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f243effd640 10 client.4312.objecter ms_dispatch 0x7f2450000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f243effd640 10 client.4312.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f243effd640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 4 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f243effd640 15 client.4312.objecter _session_command_op_remove 2 1 2024-07-27T14:41:24.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1198674224 <== osd.5 v2:172.21.15.112:6810/3751677016 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3f841e93d0 con 0x7f3f64074000 2024-07-27T14:41:24.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f3f72ffd640 10 client.4277.objecter ms_dispatch 0x7f3f84000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f3f72ffd640 10 client.4277.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f3f72ffd640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 4 2024-07-27T14:41:24.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.439+0000 7f3f72ffd640 15 client.4277.objecter _session_command_op_remove 5 2 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.441+0000 7f3f8359e640 10 librados: watch_flush enter 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.441+0000 7f3f8359e640 10 librados: watch_flush exit 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.441+0000 7f3f8359e640 10 client.4277.objecter close_session for osd.5 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f3f64074000 msgr2=0x7f3f64076440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f3f64074000 0x7f3f64076440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 20 client.4277.objecter put_session s=0x7f3f64073930 osd=5 1 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 20 client.4277.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 10 client.4277.objecter successfully canceled tick 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f6406dbc0 msgr2=0x7f3f64070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f6406dbc0 0x7f3f64070080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f74036000 tx=0x7f3f74030000 comp rx=0 tx=0).stop 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 10 monclient: shutdown 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 20 monclient: shutdown canceling and discarding version request 1 2024-07-27T14:41:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 msgr2=0x7f3f84072030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 --2- 172.21.15.87:0/1198674224 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f840b0720 0x7f3f84072030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800c3f0 tx=0x7f3f7800c8b0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f82d9d640 1 -- 172.21.15.87:0/1198674224 reap_dead start 2024-07-27T14:41:24.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 shutdown_connections 2024-07-27T14:41:24.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.442+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 >> 172.21.15.87:0/1198674224 conn(0x7f3f840b4c30 msgr2=0x7f3f8414e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.446+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 shutdown_connections 2024-07-27T14:41:24.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.446+0000 7f3f8359e640 1 -- 172.21.15.87:0/1198674224 wait complete. 2024-07-27T14:41:24.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.446+0000 7f3f8359e640 1 librados: shutdown 2024-07-27T14:41:24.456 INFO:teuthology.orchestra.run.smithi087.stdout:55834574854 2024-07-27T14:41:24.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-27T14:41:24.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.461+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== osd.4 v2:172.21.15.112:6826/1578415927 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e08076790 con 0x7f1e08073c80 2024-07-27T14:41:24.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.461+0000 7f1e16ffd640 10 client.4280.objecter ms_dispatch 0x7f1e28000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.461+0000 7f1e16ffd640 10 client.4280.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.461+0000 7f1e16ffd640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 4 2024-07-27T14:41:24.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.461+0000 7f1e16ffd640 15 client.4280.objecter _session_command_op_remove 4 1 2024-07-27T14:41:24.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.462+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b98054f30 con 0x7f9ba80734a0 2024-07-27T14:41:24.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.463+0000 7f9b9effd640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 15 client.4318.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 20 client.4318.objecter _get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 20 client.4318.objecter _get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 15 client.4318.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 20 client.4318.objecter get_session s=0x7f9b8c073530 osd=1 3 2024-07-27T14:41:24.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 15 client.4318.objecter _session_command_op_assign 1 2 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 4 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 10 client.4318.objecter _send_command 2 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.467+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 --> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ba813f580 con 0x7f9b8c073c00 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 15 client.4312.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 20 client.4312.objecter _get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 20 client.4312.objecter _get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 15 client.4312.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 20 client.4312.objecter get_session s=0x7f24300735b0 osd=2 3 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 15 client.4312.objecter _session_command_op_assign 2 2 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 4 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 10 client.4312.objecter _send_command 2 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f245013f1e0 con 0x7f2430073c80 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 15 client.4315.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 20 client.4315.objecter _get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 20 client.4315.objecter _get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 15 client.4315.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 20 client.4315.objecter get_session s=0x7f62fc073580 osd=6 3 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 15 client.4315.objecter _session_command_op_assign 6 2 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 4 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 10 client.4315.objecter _send_command 2 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.468+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 --> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f631806d2c0 con 0x7f62fc073c50 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.469+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2440077470 con 0x7f24500aad90 2024-07-27T14:41:24.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.470+0000 7f243effd640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.470+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6310055410 con 0x7f631813c320 2024-07-27T14:41:24.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.470+0000 7f63157fa640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.475+0000 7f9b9effd640 1 -- 172.21.15.87:0/3063704240 <== osd.1 v2:172.21.15.87:6800/1509783986 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9ba813f580 con 0x7f9b8c073c00 2024-07-27T14:41:24.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.475+0000 7f9b9effd640 10 client.4318.objecter ms_dispatch 0x7f9ba8000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.475+0000 7f9b9effd640 10 client.4318.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.475+0000 7f9b9effd640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 4 2024-07-27T14:41:24.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.475+0000 7f9b9effd640 15 client.4318.objecter _session_command_op_remove 1 2 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 10 librados: watch_flush enter 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 10 librados: watch_flush exit 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 10 client.4318.objecter close_session for osd.1 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7f9b8c073c00 msgr2=0x7f9b8c076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7f9b8c073c00 0x7f9b8c076040 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 20 client.4318.objecter put_session s=0x7f9b8c073530 osd=1 1 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 20 client.4318.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 10 client.4318.objecter successfully canceled tick 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9b8c06d7c0 msgr2=0x7f9b8c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f9b8c06d7c0 0x7f9b8c06fc80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80b4350 tx=0x7f9ba4006bf0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 10 monclient: shutdown 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 msgr2=0x7f9ba80b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 --2- 172.21.15.87:0/3063704240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba80734a0 0x7f9ba80b3b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c4d0 tx=0x7f9b9800c990 comp rx=0 tx=0).stop 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bae9b7640 1 -- 172.21.15.87:0/3063704240 reap_dead start 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 shutdown_connections 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 >> 172.21.15.87:0/3063704240 conn(0x7f9ba80b49f0 msgr2=0x7f9ba813e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 shutdown_connections 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.476+0000 7f9bb041a640 1 -- 172.21.15.87:0/3063704240 wait complete. 2024-07-27T14:41:24.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.477+0000 7f9bb041a640 1 librados: shutdown 2024-07-27T14:41:24.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.479+0000 7f63157fa640 1 -- 172.21.15.87:0/3062651952 <== osd.6 v2:172.21.15.112:6802/591715754 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f631806d2c0 con 0x7f62fc073c50 2024-07-27T14:41:24.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.479+0000 7f63157fa640 10 client.4315.objecter ms_dispatch 0x7f6318000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.479+0000 7f63157fa640 10 client.4315.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.479+0000 7f63157fa640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 4 2024-07-27T14:41:24.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f63157fa640 15 client.4315.objecter _session_command_op_remove 6 2 2024-07-27T14:41:24.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 10 librados: watch_flush enter 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 10 librados: watch_flush exit 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 10 client.4315.objecter close_session for osd.6 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f62fc073c50 msgr2=0x7f62fc076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f62fc073c50 0x7f62fc076090 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 20 client.4315.objecter put_session s=0x7f62fc073580 osd=6 1 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 20 client.4315.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 10 client.4315.objecter successfully canceled tick 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f62fc06d810 msgr2=0x7f62fc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f62fc06d810 0x7f62fc06fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f630c036000 tx=0x7f630c030000 comp rx=0 tx=0).stop 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 10 monclient: shutdown 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 msgr2=0x7f6318140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631def0640 1 --2- 172.21.15.87:0/3062651952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f631813c320 0x7f6318140350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6310007ad0 tx=0x7f6310007f90 comp rx=0 tx=0).stop 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.480+0000 7f631d6ef640 1 -- 172.21.15.87:0/3062651952 reap_dead start 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 shutdown_connections 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 >> 172.21.15.87:0/3062651952 conn(0x7f63180b4c30 msgr2=0x7f63180b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 shutdown_connections 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f631def0640 1 -- 172.21.15.87:0/3062651952 wait complete. 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f631def0640 1 librados: shutdown 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f243effd640 1 -- 172.21.15.87:0/860260179 <== osd.2 v2:172.21.15.87:6820/357294290 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f245013f1e0 con 0x7f2430073c80 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f243effd640 10 client.4312.objecter ms_dispatch 0x7f2450000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f243effd640 10 client.4312.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f243effd640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 4 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.481+0000 7f243effd640 15 client.4312.objecter _session_command_op_remove 2 2 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 10 librados: watch_flush enter 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 10 librados: watch_flush exit 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 10 client.4312.objecter close_session for osd.2 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f2430073c80 msgr2=0x7f24300760c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f2430073c80 0x7f24300760c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 20 client.4312.objecter put_session s=0x7f24300735b0 osd=2 1 2024-07-27T14:41:24.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f244f59e640 10 client.4312.objecter successfully canceled tick 2024-07-27T14:41:24.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f243006d840 msgr2=0x7f243006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f243006d840 0x7f243006fd00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2448007250 tx=0x7f244800f040 comp rx=0 tx=0).stop 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 10 monclient: shutdown 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 msgr2=0x7f24501d4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 --2- 172.21.15.87:0/860260179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24500aad90 0x7f24501d4000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2440009890 tx=0x7f24400046c0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244ed9d640 1 -- 172.21.15.87:0/860260179 reap_dead start 2024-07-27T14:41:24.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 15 client.4280.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 shutdown_connections 2024-07-27T14:41:24.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 >> 172.21.15.87:0/860260179 conn(0x7f2450067e10 msgr2=0x7f2450136c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 20 client.4280.objecter _get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 20 client.4280.objecter _get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 15 client.4280.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 20 client.4280.objecter get_session s=0x7f1e080735b0 osd=4 3 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 15 client.4280.objecter _session_command_op_assign 4 2 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 4 2024-07-27T14:41:24.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 10 client.4280.objecter _send_command 2 2024-07-27T14:41:24.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.482+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 --> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e281d87c0 con 0x7f1e08073c80 2024-07-27T14:41:24.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.483+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faa6801a070 con 0x7faa740728e0 2024-07-27T14:41:24.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 shutdown_connections 2024-07-27T14:41:24.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7f244f59e640 1 -- 172.21.15.87:0/860260179 wait complete. 2024-07-27T14:41:24.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7faa65ffb640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faa5806d810 0x7faa5806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7f244f59e640 1 librados: shutdown 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7faa78b9d640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faa5806d810 0x7faa5806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.484+0000 7faa78b9d640 10 monclient: get_auth_request con 0x7faa5806d810 auth_method 0 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.485+0000 7faa78b9d640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faa5806d810 0x7faa5806fcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa70000f60 tx=0x7faa70007a80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.485+0000 7faa65ffb640 10 client.4324.objecter ms_handle_connect 0x7faa5806d810 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7faa68056d70 con 0x7faa740728e0 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 10 client.4324.objecter ms_dispatch 0x7faa74000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 3 client.4324.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 10 client.4324.objecter checking command 1 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7faa58073c50 0x7faa58076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter _get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 15 client.4324.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter _get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 15 client.4324.objecter _session_command_op_assign 3 1 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 4 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 10 client.4324.objecter _send_command 1 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa58076760 con 0x7faa58073c50 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faa68054f70 con 0x7faa740728e0 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.486+0000 7faa65ffb640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.487+0000 7faa79b9f640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7faa58073c50 0x7faa58076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.487+0000 7faa79b9f640 10 monclient: get_auth_request con 0x7faa58073c50 auth_method 0 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f1e16ffd640 1 -- 172.21.15.87:0/1139173911 <== osd.4 v2:172.21.15.112:6826/1578415927 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1e281d87c0 con 0x7f1e08073c80 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f1e16ffd640 10 client.4280.objecter ms_dispatch 0x7f1e28000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f1e16ffd640 10 client.4280.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f1e16ffd640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 4 2024-07-27T14:41:24.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f1e16ffd640 15 client.4280.objecter _session_command_op_remove 4 2 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8017070 con 0x7f27b402be60 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: dump: 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.495 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a800ee60 con 0x7f27b402be60 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a800cb40 con 0x7f27b402be60 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27a7fff640 10 monclient: dump: 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.496 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.490+0000 7f27b9ffb640 4 set_mon_vals no callback set 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 5 monclient: authenticate success, global_id 4283 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 10 monclient: _renew_subs 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b40a0850 con 0x7f27b402be60 2024-07-27T14:41:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 10 monclient: _renew_subs 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b402a360 con 0x7f27b402be60 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.491+0000 7f27c0c4d640 1 librados: init done 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 15 client.4283.objecter _session_command_op_assign -1 1 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 20 client.4283.objecter _get_session osd=-1 returning homeless 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27b40388f0 con 0x7f27b402be60 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7faa79b9f640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7faa58073c50 0x7faa58076090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.492+0000 7faa65ffb640 10 client.4324.objecter ms_handle_connect 0x7faa58073c50 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 10 librados: watch_flush enter 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 10 librados: watch_flush exit 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 10 client.4280.objecter close_session for osd.4 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f1e08073c80 msgr2=0x7f1e080760c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f1e08073c80 0x7f1e080760c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 20 client.4280.objecter put_session s=0x7f1e080735b0 osd=4 1 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 20 client.4280.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 10 client.4280.objecter successfully canceled tick 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.493+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e0806d840 msgr2=0x7f1e0806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f1e0806d840 0x7f1e0806fd00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e18030000 comp rx=0 tx=0).stop 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 10 monclient: shutdown 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 20 monclient: shutdown canceling and discarding version request 1 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 msgr2=0x7f1e281d8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 --2- 172.21.15.87:0/1139173911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1e280af970 0x7f1e281d8b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b450 tx=0x7f1e1c00b910 comp rx=0 tx=0).stop 2024-07-27T14:41:24.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e26d9d640 1 -- 172.21.15.87:0/1139173911 reap_dead start 2024-07-27T14:41:24.501 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-27T14:41:24.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 shutdown_connections 2024-07-27T14:41:24.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 >> 172.21.15.87:0/1139173911 conn(0x7f1e28068150 msgr2=0x7f1e280b7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 shutdown_connections 2024-07-27T14:41:24.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 -- 172.21.15.87:0/1139173911 wait complete. 2024-07-27T14:41:24.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.494+0000 7f1e2ca0e640 1 librados: shutdown 2024-07-27T14:41:24.503 INFO:teuthology.orchestra.run.smithi087.stdout:77309411337 2024-07-27T14:41:24.503 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-27T14:41:24.504 INFO:teuthology.orchestra.run.smithi087.stdout:77309411336 2024-07-27T14:41:24.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-27T14:41:24.507 INFO:teuthology.orchestra.run.smithi087.stdout:55834574855 2024-07-27T14:41:24.507 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-07-27T14:41:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.528+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff9cc01a070 con 0x7ff9d80b1b80 2024-07-27T14:41:24.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.529+0000 7ff9d4ff9640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff9c406d7c0 0x7ff9c406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.529+0000 7ff9d7fff640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff9c406d7c0 0x7ff9c406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.529+0000 7ff9d7fff640 10 monclient: get_auth_request con 0x7ff9c406d7c0 auth_method 0 2024-07-27T14:41:24.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.530+0000 7ff9d4ff9640 10 client.4286.objecter ms_handle_connect 0x7ff9c406d7c0 2024-07-27T14:41:24.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.530+0000 7ff9d7fff640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff9c406d7c0 0x7ff9c406fc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.546+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f27a801a070 con 0x7f27b402be60 2024-07-27T14:41:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.547+0000 7f27a7fff640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f27a006d7c0 0x7f27a006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.548+0000 7f27bb7fe640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f27a006d7c0 0x7f27a006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.548+0000 7f27bb7fe640 10 monclient: get_auth_request con 0x7f27a006d7c0 auth_method 0 2024-07-27T14:41:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.548+0000 7f27bb7fe640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f27a006d7c0 0x7f27a006fc80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27b0036000 tx=0x7f27b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.548+0000 7f27a7fff640 10 client.4283.objecter ms_handle_connect 0x7f27a006d7c0 2024-07-27T14:41:24.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f27a8057a20 con 0x7f27b402be60 2024-07-27T14:41:24.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 10 client.4283.objecter ms_dispatch 0x7f27bc000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 3 client.4283.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 3 client.4283.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 10 client.4283.objecter checking command 1 2024-07-27T14:41:24.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f27a0073c00 0x7f27a0076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter _get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 15 client.4283.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter _get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 15 client.4283.objecter _session_command_op_assign 0 1 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 4 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 10 client.4283.objecter _send_command 1 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27a0076710 con 0x7f27a0073c00 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 20 client.4283.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f27a8055c20 con 0x7f27b402be60 2024-07-27T14:41:24.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.557+0000 7f27a7fff640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7ff9cc056d10 con 0x7ff9d80b1b80 2024-07-27T14:41:24.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 10 client.4286.objecter ms_dispatch 0x7ff9d8000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 3 client.4286.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 3 client.4286.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 10 client.4286.objecter checking command 1 2024-07-27T14:41:24.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7ff9c4073c00 0x7ff9c4076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter _get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 15 client.4286.objecter _session_command_op_remove -1 1 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter _get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 15 client.4286.objecter _session_command_op_assign 7 1 2024-07-27T14:41:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 4 2024-07-27T14:41:24.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 10 client.4286.objecter _send_command 1 2024-07-27T14:41:24.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9c4076710 con 0x7ff9c4073c00 2024-07-27T14:41:24.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 20 client.4286.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff9cc054f10 con 0x7ff9d80b1b80 2024-07-27T14:41:24.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7ff9d4ff9640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-27T14:41:24.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7f27bbfff640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f27a0073c00 0x7f27a0076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.559+0000 7f27bbfff640 10 monclient: get_auth_request con 0x7f27a0073c00 auth_method 0 2024-07-27T14:41:24.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.560+0000 7ff9dcdff640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7ff9c4073c00 0x7ff9c4076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.560+0000 7ff9dcdff640 10 monclient: get_auth_request con 0x7ff9c4073c00 auth_method 0 2024-07-27T14:41:24.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.563+0000 7ff9dcdff640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7ff9c4073c00 0x7ff9c4076040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.563+0000 7ff9d4ff9640 10 client.4286.objecter ms_handle_connect 0x7ff9c4073c00 2024-07-27T14:41:24.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.564+0000 7f27bbfff640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f27a0073c00 0x7f27a0076040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.566+0000 7f27a7fff640 10 client.4283.objecter ms_handle_connect 0x7f27a0073c00 2024-07-27T14:41:24.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.571+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== osd.3 v2:172.21.15.87:6816/1100053525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa58076760 con 0x7faa58073c50 2024-07-27T14:41:24.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.571+0000 7faa65ffb640 10 client.4324.objecter ms_dispatch 0x7faa74000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.571+0000 7faa65ffb640 10 client.4324.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.571+0000 7faa65ffb640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 4 2024-07-27T14:41:24.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.571+0000 7faa65ffb640 15 client.4324.objecter _session_command_op_remove 3 1 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 15 client.4324.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 20 client.4324.objecter _get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 20 client.4324.objecter _get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 15 client.4324.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 20 client.4324.objecter get_session s=0x7faa58073580 osd=3 3 2024-07-27T14:41:24.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 15 client.4324.objecter _session_command_op_assign 3 2 2024-07-27T14:41:24.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 4 2024-07-27T14:41:24.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 10 client.4324.objecter _send_command 2 2024-07-27T14:41:24.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.601+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa740b4f40 con 0x7faa58073c50 2024-07-27T14:41:24.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== osd.7 v2:172.21.15.112:6818/2667521237 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9c4076710 con 0x7ff9c4073c00 2024-07-27T14:41:24.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7ff9d4ff9640 10 client.4286.objecter ms_dispatch 0x7ff9d8000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7ff9d4ff9640 10 client.4286.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7ff9d4ff9640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 4 2024-07-27T14:41:24.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7ff9d4ff9640 15 client.4286.objecter _session_command_op_remove 7 1 2024-07-27T14:41:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7faa65ffb640 1 -- 172.21.15.87:0/2042148412 <== osd.3 v2:172.21.15.87:6816/1100053525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faa740b4f40 con 0x7faa58073c50 2024-07-27T14:41:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7faa65ffb640 10 client.4324.objecter ms_dispatch 0x7faa74000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7faa65ffb640 10 client.4324.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7faa65ffb640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 4 2024-07-27T14:41:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.609+0000 7faa65ffb640 15 client.4324.objecter _session_command_op_remove 3 2 2024-07-27T14:41:24.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.610+0000 7faa7a3a0640 10 librados: watch_flush enter 2024-07-27T14:41:24.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.610+0000 7faa7a3a0640 10 librados: watch_flush exit 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.610+0000 7faa7a3a0640 10 client.4324.objecter close_session for osd.3 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.611+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7faa58073c50 msgr2=0x7faa58076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.611+0000 7faa7a3a0640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7faa58073c50 0x7faa58076090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.613+0000 7faa7a3a0640 20 client.4324.objecter put_session s=0x7faa58073580 osd=3 1 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.613+0000 7faa7a3a0640 20 client.4324.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.613+0000 7faa7a3a0640 10 client.4324.objecter successfully canceled tick 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faa5806d810 msgr2=0x7faa5806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faa5806d810 0x7faa5806fcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa70000f60 tx=0x7faa70007a80 comp rx=0 tx=0).stop 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 10 monclient: shutdown 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 msgr2=0x7faa7413c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 --2- 172.21.15.87:0/2042148412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa740728e0 0x7faa7413c690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c410 tx=0x7faa6800c8d0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa79b9f640 1 -- 172.21.15.87:0/2042148412 reap_dead start 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 shutdown_connections 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.614+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 >> 172.21.15.87:0/2042148412 conn(0x7faa740682b0 msgr2=0x7faa740b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.615+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 shutdown_connections 2024-07-27T14:41:24.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.615+0000 7faa7a3a0640 1 -- 172.21.15.87:0/2042148412 wait complete. 2024-07-27T14:41:24.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.615+0000 7faa7a3a0640 1 librados: shutdown 2024-07-27T14:41:24.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.622+0000 7f6bed78c640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.622+0000 7f6bed78c640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.622+0000 7f6bed78c640 10 monclient: monmap: 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.623388+0000 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.623388+0000 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.630 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.623+0000 7f6bed78c640 1 Processor -- start 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.623+0000 7f6bed78c640 1 -- start start 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.623+0000 7f6bed78c640 10 monclient: init 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6be80b1a80 0x7f6be80aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-a con 0x7f6be80b1a80 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6be80ab2e0 0x7f6be80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-c con 0x7f6be80ab2e0 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6be80b0550 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6be80ad960 con 0x7f6be80b0550 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f6be80ab730 con 0x7f6be80b1a80 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f6be80abb80 con 0x7f6be80ab2e0 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.624+0000 7f6bed78c640 10 monclient(hunting): authenticate will time out at 1224.650736s 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.626+0000 7f6be7fff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6be80b1a80 0x7f6be80aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.626+0000 7f6be7fff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6be80b1a80 0x7f6be80aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42772/0 (socket says 172.21.15.87:42772) 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be7fff640 1 -- 172.21.15.87:0/2217237945 learned_addr learned my addr 172.21.15.87:0/2217237945 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be7fff640 10 monclient(hunting): get_auth_request con 0x7f6be80b1a80 auth_method 0 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be77fe640 1 --2- 172.21.15.87:0/2217237945 >> v2:172.21.15.112:3300/0 conn(0x7f6be80ab2e0 0x7f6be80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be77fe640 10 monclient(hunting): get_auth_request con 0x7f6be80ab2e0 auth_method 0 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be77fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.627+0000 7f6be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.629+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.636 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:24.637 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-27T14:41:24.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-07-27T14:41:24.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 1 -- 172.21.15.87:0/2217237945 >> v2:172.21.15.112:3300/0 conn(0x7f6be80ab2e0 msgr2=0x7f6be80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2217237945 >> v2:172.21.15.112:3300/0 conn(0x7f6be80ab2e0 0x7f6be80b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 1 -- 172.21.15.87:0/2217237945 >> v1:172.21.15.112:6789/0 conn(0x7f6be80b0550 legacy=0x7f6be8159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 1 monclient: found mon.noname-a 2024-07-27T14:41:24.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.633+0000 7f6be7fff640 1 -- 172.21.15.87:0/2217237945 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be80ad540 con 0x7f6be80b1a80 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 15 client.4286.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 20 client.4286.objecter _get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 20 client.4286.objecter _get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 15 client.4286.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 20 client.4286.objecter get_session s=0x7ff9c4073530 osd=7 3 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 15 client.4286.objecter _session_command_op_assign 7 2 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 4 2024-07-27T14:41:24.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 10 client.4286.objecter _send_command 2 2024-07-27T14:41:24.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.635+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 --> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff9d8136070 con 0x7ff9c4073c00 2024-07-27T14:41:24.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.636+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2217237945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be80aada0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6be80b2030 tx=0x7f6bd8008230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c43cc14bd09954 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 1 -- 172.21.15.87:0/2217237945 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8030bf0 con 0x7f6be80b1a80 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 10 monclient: dump: 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6bed78c640 5 monclient: authenticate success, global_id 4305 2024-07-27T14:41:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7f6bed78c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7ff9d4ff9640 1 -- 172.21.15.87:0/4223377180 <== osd.7 v2:172.21.15.112:6818/2667521237 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff9d8136070 con 0x7ff9c4073c00 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7ff9d4ff9640 10 client.4286.objecter ms_dispatch 0x7ff9d8000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7ff9d4ff9640 10 client.4286.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7ff9d4ff9640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 4 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.641+0000 7ff9d4ff9640 15 client.4286.objecter _session_command_op_remove 7 2 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 10 librados: watch_flush enter 2024-07-27T14:41:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 10 librados: watch_flush exit 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 10 client.4286.objecter close_session for osd.7 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7ff9c4073c00 msgr2=0x7ff9c4076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7ff9c4073c00 0x7ff9c4076040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 20 client.4286.objecter put_session s=0x7ff9c4073530 osd=7 1 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 20 client.4286.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 10 client.4286.objecter successfully canceled tick 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff9c406d7c0 msgr2=0x7ff9c406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff9c406d7c0 0x7ff9c406fc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0036000 tx=0x7ff9d002e000 comp rx=0 tx=0).stop 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 10 monclient: shutdown 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 msgr2=0x7ff9d8139480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9de862640 1 --2- 172.21.15.87:0/4223377180 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff9d80b1b80 0x7ff9d8139480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c430 tx=0x7ff9cc00c8f0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.642+0000 7ff9dcdff640 1 -- 172.21.15.87:0/4223377180 reap_dead start 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.643+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 shutdown_connections 2024-07-27T14:41:24.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.643+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 >> 172.21.15.87:0/4223377180 conn(0x7ff9d80b4d30 msgr2=0x7ff9d80b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.643+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 shutdown_connections 2024-07-27T14:41:24.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.643+0000 7ff9de862640 1 -- 172.21.15.87:0/4223377180 wait complete. 2024-07-27T14:41:24.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.643+0000 7ff9de862640 1 librados: shutdown 2024-07-27T14:41:24.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.644+0000 7f6be6ffd640 1 -- 172.21.15.87:0/2217237945 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd800ae50 con 0x7f6be80b1a80 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.644+0000 7f6be6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.644+0000 7f6bed78c640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.644+0000 7f6bed78c640 4 set_mon_vals no callback set 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.645+0000 7f6bed78c640 10 monclient: shutdown 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.645+0000 7f6bed78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.645+0000 7f6bed78c640 1 -- 172.21.15.87:0/2217237945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 msgr2=0x7f6be80aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.645+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2217237945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be80aada0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6be80b2030 tx=0x7f6bd8008230 comp rx=0 tx=0).stop 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.646+0000 7f6bed78c640 1 -- 172.21.15.87:0/2217237945 shutdown_connections 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.646+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2217237945 >> v2:172.21.15.112:3300/0 conn(0x7f6be80ab2e0 0x7f6be80b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.646+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2217237945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be80aada0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.646+0000 7f6bed78c640 1 -- 172.21.15.87:0/2217237945 >> 172.21.15.87:0/2217237945 conn(0x7f6be80b4c30 msgr2=0x7f6be80b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== osd.0 v2:172.21.15.87:6808/3385608780 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27a0076710 con 0x7f27a0073c00 2024-07-27T14:41:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f27a7fff640 10 client.4283.objecter ms_dispatch 0x7f27bc000b90 command_reply(tid 1: 0 ) 2024-07-27T14:41:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f27a7fff640 10 client.4283.objecter _finish_command 1 = system:0 2024-07-27T14:41:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f27a7fff640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 4 2024-07-27T14:41:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f27a7fff640 15 client.4283.objecter _session_command_op_remove 0 1 2024-07-27T14:41:24.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f6bed78c640 1 -- 172.21.15.87:0/2217237945 shutdown_connections 2024-07-27T14:41:24.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.649+0000 7f6bed78c640 1 -- 172.21.15.87:0/2217237945 wait complete. 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.650+0000 7f6bed78c640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.650+0000 7f6bed78c640 10 monclient: monmap: 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.650+0000 7f6bed78c640 1 librados: starting msgr at 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.650+0000 7f6bed78c640 1 librados: starting objecter 2024-07-27T14:41:24.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.650+0000 7f6bed78c640 1 Processor -- start 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.651+0000 7f6bed78c640 1 -- start start 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.651+0000 7f6bed78c640 1 librados: setting wanted keys 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.651+0000 7f6bed78c640 1 librados: calling monclient init 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.651+0000 7f6bed78c640 10 monclient: init 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 0x7f6be8085490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-b con 0x7f6be80ab2e0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6be80b0550 0x7f6be80859d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-c con 0x7f6be80b0550 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): picked mon.noname-a con 0x7f6be80b1a80 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6be808cc20 con 0x7f6be80b1a80 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6be80abea0 con 0x7f6be80ab2e0 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f6be808cda0 con 0x7f6be80b0550 2024-07-27T14:41:24.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.652+0000 7f6bed78c640 10 monclient(hunting): authenticate will time out at 1224.678339s 2024-07-27T14:41:24.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.654+0000 7f6becf8b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42782/0 (socket says 172.21.15.87:42782) 2024-07-27T14:41:24.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 1 -- 172.21.15.87:0/2703091661 learned_addr learned my addr 172.21.15.87:0/2703091661 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 10 monclient(hunting): get_auth_request con 0x7f6be80b1a80 auth_method 0 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6becf8b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 0x7f6be8085490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be77fe640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6be80b0550 0x7f6be80859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be77fe640 10 monclient(hunting): get_auth_request con 0x7f6be80b0550 auth_method 0 2024-07-27T14:41:24.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be77fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.655+0000 7f6be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be7fff640 10 monclient(hunting): get_auth_request con 0x7f6be80ab2e0 auth_method 0 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.656+0000 7f6be77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.657+0000 7f6becf8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.657+0000 7f6becf8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.657+0000 7f6becf8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.660+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.660+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.660+0000 7f6be7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.663 INFO:teuthology.orchestra.run.smithi087.stdout:55834574855 2024-07-27T14:41:24.664 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-27T14:41:24.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.663+0000 7f6be7fff640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-07-27T14:41:24.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.663+0000 7f6be7fff640 1 -- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6be80b0550 msgr2=0x7f6be80859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.663+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6be80b0550 0x7f6be80859d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.663+0000 7f6be7fff640 1 -- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 msgr2=0x7f6be81f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6be7fff640 1 -- 172.21.15.87:0/2703091661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be80867e0 con 0x7f6be80ab2e0 2024-07-27T14:41:24.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.664+0000 7f6becf8b640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:24.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.669+0000 7f6be7fff640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 0x7f6be8085490 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8036000 tx=0x7f6bd8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.669+0000 7f6be4ff9640 10 client.?.objecter ms_handle_connect 0x7f6be80ab2e0 2024-07-27T14:41:24.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.669+0000 7f6be4ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8047070 con 0x7f6be80ab2e0 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: dump: 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.673 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd8042630 con 0x7f6be80ab2e0 2024-07-27T14:41:24.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be4ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.672+0000 7f6be5ffb640 4 set_mon_vals no callback set 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 5 monclient: authenticate success, global_id 4330 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 10 monclient: _renew_subs 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be8086ac0 con 0x7f6be80ab2e0 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 10 monclient: _renew_subs 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be81f14e0 con 0x7f6be80ab2e0 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6bed78c640 1 librados: init done 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8042a20 con 0x7f6be80ab2e0 2024-07-27T14:41:24.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6be4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6be4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.673+0000 7f6be4ff9640 10 monclient: dump: 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.674+0000 7f6bed78c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.675+0000 7f6bed78c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.675+0000 7f6bed78c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.675+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be804fd50 con 0x7f6be80ab2e0 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.677+0000 7f8a3e90f640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.677+0000 7f8a3e90f640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.677+0000 7f8a3e90f640 10 monclient: monmap: 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.678621+0000 2024-07-27T14:41:24.678 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.678621+0000 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f8a3e90f640 1 Processor -- start 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 15 client.4283.objecter _session_command_op_assign -1 2 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 20 client.4283.objecter _get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 20 client.4283.objecter _get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 15 client.4283.objecter _session_command_op_remove -1 2 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 20 client.4283.objecter get_session s=0x7f27a0073530 osd=0 3 2024-07-27T14:41:24.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 15 client.4283.objecter _session_command_op_assign 0 2 2024-07-27T14:41:24.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 4 2024-07-27T14:41:24.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 10 client.4283.objecter _send_command 2 2024-07-27T14:41:24.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 --> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27b4007720 con 0x7f27a0073c00 2024-07-27T14:41:24.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f8a3e90f640 1 -- start start 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.678+0000 7f8a3e90f640 10 monclient: init 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a30006c00 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 0x7f8a30030830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3002be70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a300315b0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a30032c60 con 0x7f8a300315b0 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a3002c350 con 0x7f8a30006c00 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f8a30006fe0 con 0x7f8a3002be70 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.679+0000 7f8a3e90f640 10 monclient(hunting): authenticate will time out at 1224.705910s 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3d10c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 0x7f8a30030830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3e10e640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f8a300315b0 0x7f8a300d6190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45262/0 (socket says 172.21.15.87:45262) 2024-07-27T14:41:24.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3e10e640 1 -- 172.21.15.87:0/3342640782 learned_addr learned my addr 172.21.15.87:0/3342640782 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3d10c640 10 monclient(hunting): get_auth_request con 0x7f8a3002be70 auth_method 0 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3d10c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3d10c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.681+0000 7f8a3d10c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.683+0000 7f8a3d10c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.683+0000 7f8a3d10c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.683+0000 7f8a3d10c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.685+0000 7f8a3d10c640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-07-27T14:41:24.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 1 -- 172.21.15.87:0/3342640782 >> v1:172.21.15.112:6789/0 conn(0x7f8a30006c00 legacy=0x7f8a3002f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 1 -- 172.21.15.87:0/3342640782 >> v1:172.21.15.87:6789/0 conn(0x7f8a300315b0 legacy=0x7f8a300d6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3c90b640 1 -- 172.21.15.87:0/3342640782 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277589739 0 0) 0x7f8a30032c60 con 0x7f8a300315b0 2024-07-27T14:41:24.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3d10c640 1 -- 172.21.15.87:0/3342640782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a30034480 con 0x7f8a3002be70 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.686+0000 7f8a3c90b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f27a7fff640 1 -- 172.21.15.87:0/3843126661 <== osd.0 v2:172.21.15.87:6808/3385608780 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f27b4007720 con 0x7f27a0073c00 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f27a7fff640 10 client.4283.objecter ms_dispatch 0x7f27bc000b90 command_reply(tid 2: 0 ) 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f27a7fff640 10 client.4283.objecter _finish_command 2 = system:0 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.688+0000 7f27a7fff640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 4 2024-07-27T14:41:24.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.688+0000 7f27a7fff640 15 client.4283.objecter _session_command_op_remove 0 2 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f2ac1b45640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f2ac1b45640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.687+0000 7f2ac1b45640 10 monclient: monmap: 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.688287+0000 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.688287+0000 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.689 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.688+0000 7f2ac1b45640 1 Processor -- start 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.688+0000 7f2ac1b45640 1 -- start start 2024-07-27T14:41:24.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.688+0000 7f2ac1b45640 10 monclient: init 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 10 librados: watch_flush enter 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 10 librados: watch_flush exit 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 10 client.4283.objecter close_session for osd.0 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f27a0073c00 msgr2=0x7f27a0076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f27a0073c00 0x7f27a0076040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 20 client.4283.objecter put_session s=0x7f27a0073530 osd=0 1 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 20 client.4283.objecter shutdown clearing up homeless session... 2024-07-27T14:41:24.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f27c0c4d640 10 client.4283.objecter successfully canceled tick 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2abc0abed0 0x7f2abc0ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-c con 0x7f2abc0abed0 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 0x7f2abc0ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-b con 0x7f2abc0735e0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2abc0adf80 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc0ad3b0 con 0x7f2abc0adf80 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f2abc0aed20 con 0x7f2abc0abed0 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc144b70 con 0x7f2abc0735e0 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.689+0000 7f2ac1b45640 10 monclient(hunting): authenticate will time out at 1224.715522s 2024-07-27T14:41:24.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.690+0000 7f8a3d10c640 1 --2- 172.21.15.87:0/3342640782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 0x7f8a30030830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097d0 tx=0x7f8a2802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5790c6246567140 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.690+0000 7f2abaffd640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 0x7f2abc0ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.690+0000 7f2abbfff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f2abc0adf80 0x7f2abc144460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45266/0 (socket says 172.21.15.87:45266) 2024-07-27T14:41:24.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.690+0000 7f2abbfff640 1 -- 172.21.15.87:0/500459218 learned_addr learned my addr 172.21.15.87:0/500459218 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f27a006d7c0 msgr2=0x7f27a006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f27a006d7c0 0x7f27a006fc80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27b0036000 tx=0x7f27b0030000 comp rx=0 tx=0).stop 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 10 monclient: shutdown 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 msgr2=0x7f27b4161210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 --2- 172.21.15.87:0/3843126661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f27b402be60 0x7f27b4161210 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f27a800e870 tx=0x7f27a800ed30 comp rx=0 tx=0).stop 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27bbfff640 1 -- 172.21.15.87:0/3843126661 reap_dead start 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 shutdown_connections 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 >> 172.21.15.87:0/3843126661 conn(0x7f27b4005a40 msgr2=0x7f27b402cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f2abaffd640 10 monclient(hunting): get_auth_request con 0x7f2abc0735e0 auth_method 0 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f2abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f2abaffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.691+0000 7f2abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.692+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 shutdown_connections 2024-07-27T14:41:24.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.692+0000 7f27c0c4d640 1 -- 172.21.15.87:0/3843126661 wait complete. 2024-07-27T14:41:24.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.692+0000 7f27c0c4d640 1 librados: shutdown 2024-07-27T14:41:24.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.692+0000 7fd3086cc640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.692+0000 7fd3086cc640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.693+0000 7fd3086cc640 10 monclient: monmap: 2024-07-27T14:41:24.698 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.698 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.694038+0000 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.694038+0000 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.693+0000 7fd3086cc640 1 Processor -- start 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.693+0000 7fd3086cc640 1 -- start start 2024-07-27T14:41:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.693+0000 7fd3086cc640 10 monclient: init 2024-07-27T14:41:24.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.694+0000 7f2abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.694+0000 7f2abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.694+0000 7f2abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd3000b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-b con 0x7fd300072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3000af970 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3000b0410 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3000aff60 con 0x7fd3000b0410 2024-07-27T14:41:24.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3000b3060 con 0x7fd3000af970 2024-07-27T14:41:24.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 1 -- 172.21.15.87:0/500459218 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768764117 0 0) 0x7f2abc0ad3b0 con 0x7f2abc0adf80 2024-07-27T14:41:24.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 10 monclient(hunting): _negotiate 2024-07-27T14:41:24.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 10 monclient(hunting): my global_id is 4314 2024-07-27T14:41:24.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fd300072f50 con 0x7fd300072b70 2024-07-27T14:41:24.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7fd3086cc640 10 monclient(hunting): authenticate will time out at 1224.721202s 2024-07-27T14:41:24.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f2aba7fc640 1 -- 172.21.15.87:0/500459218 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa4003520 con 0x7f2abc0adf80 2024-07-27T14:41:24.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.695+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6bd804a070 con 0x7f6be80ab2e0 2024-07-27T14:41:24.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7fd306c69640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fd3000b0410 0x7fd30013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45272/0 (socket says 172.21.15.87:45272) 2024-07-27T14:41:24.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7fd306c69640 1 -- 172.21.15.87:0/2682098729 learned_addr learned my addr 172.21.15.87:0/2682098729 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f6be4ff9640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 0x7f6bd406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f6be77fe640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 0x7f6bd406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f6be77fe640 10 monclient: get_auth_request con 0x7f6bd406d840 auth_method 0 2024-07-27T14:41:24.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.697+0000 7f6be77fe640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 0x7f6bd406fd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00b240 tx=0x7f6bdc00b3a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.697+0000 7f6be4ff9640 10 client.4330.objecter ms_handle_connect 0x7f6bd406d840 2024-07-27T14:41:24.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-07-27T14:41:24.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 1 -- 172.21.15.87:0/500459218 >> v2:172.21.15.112:3300/0 conn(0x7f2abc0abed0 msgr2=0x7f2abc0ab3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 1 --2- 172.21.15.87:0/500459218 >> v2:172.21.15.112:3300/0 conn(0x7f2abc0abed0 0x7f2abc0ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 1 -- 172.21.15.87:0/500459218 >> v1:172.21.15.87:6789/0 conn(0x7f2abc0adf80 legacy=0x7f2abc144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.696+0000 7f2abaffd640 1 -- 172.21.15.87:0/500459218 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc144d20 con 0x7f2abc0735e0 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.700+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f6bd80796e0 con 0x7f6be80ab2e0 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.700+0000 7f6be4ff9640 10 client.4330.objecter ms_dispatch 0x7f6be8000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.700+0000 7f6be4ff9640 3 client.4330.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.700+0000 7f6be4ff9640 3 client.4330.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.700+0000 7f6be4ff9640 20 client.4330.objecter dump_active .. 0 homeless 2024-07-27T14:41:24.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.701+0000 7f2abaffd640 1 --2- 172.21.15.87:0/500459218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 0x7f2abc0ada40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009d90 tx=0x7f2ab402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f73141efa900fdf server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 1 -- 172.21.15.87:0/3342640782 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28035020 con 0x7f8a3002be70 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: dump: 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.709 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 1 -- 172.21.15.87:0/3342640782 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28031830 con 0x7f8a3002be70 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 1 -- 172.21.15.87:0/3342640782 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28031c00 con 0x7f8a3002be70 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3c90b640 10 monclient: dump: 2024-07-27T14:41:24.711 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.712 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.713 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.713 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3e90f640 5 monclient: authenticate success, global_id 4292 2024-07-27T14:41:24.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3e90f640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7f8a3e90f640 4 set_mon_vals no callback set 2024-07-27T14:41:24.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921580924 0 0) 0x7fd3000aff60 con 0x7fd3000b0410 2024-07-27T14:41:24.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 10 monclient(hunting): _negotiate 2024-07-27T14:41:24.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 10 monclient(hunting): my global_id is 4317 2024-07-27T14:41:24.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.706+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec003520 con 0x7fd3000b0410 2024-07-27T14:41:24.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 10 monclient: shutdown 2024-07-27T14:41:24.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 -- 172.21.15.87:0/3342640782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 msgr2=0x7f8a30030830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/3342640782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 0x7f8a30030830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a280097d0 tx=0x7f8a2802b8d0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 -- 172.21.15.87:0/3342640782 shutdown_connections 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/3342640782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a3002be70 0x7f8a30030830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 -- 172.21.15.87:0/3342640782 >> 172.21.15.87:0/3342640782 conn(0x7f8a30005a50 msgr2=0x7f8a30005e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 -- 172.21.15.87:0/3342640782 shutdown_connections 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.707+0000 7f8a3e90f640 1 -- 172.21.15.87:0/3342640782 wait complete. 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.708+0000 7f8a3e90f640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.708+0000 7f8a3e90f640 10 monclient: monmap: 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.718 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 librados: starting msgr at 2024-07-27T14:41:24.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 librados: starting objecter 2024-07-27T14:41:24.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 Processor -- start 2024-07-27T14:41:24.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7fd306468640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd3000b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.721 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:24.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 -- start start 2024-07-27T14:41:24.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 librados: setting wanted keys 2024-07-27T14:41:24.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 1 librados: calling monclient init 2024-07-27T14:41:24.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.709+0000 7f8a3e90f640 10 monclient: init 2024-07-27T14:41:24.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-27T14:41:24.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-a con 0x7f8a30006c00 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a3002be70 0x7f8a300c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3002be70 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a300315b0 0x7f8a300c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): picked mon.noname-c con 0x7f8a300315b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8a30031df0 con 0x7f8a30006c00 2024-07-27T14:41:24.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a30031c70 con 0x7f8a3002be70 2024-07-27T14:41:24.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f8a300d78b0 con 0x7f8a300315b0 2024-07-27T14:41:24.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7f8a3e90f640 10 monclient(hunting): authenticate will time out at 1224.736567s 2024-07-27T14:41:24.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7fd306468640 10 monclient(hunting): get_auth_request con 0x7fd300072b70 auth_method 0 2024-07-27T14:41:24.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7fd306468640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7fd306468640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.710+0000 7fd306468640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1089052505 0 0) 0x7fd2ec003520 con 0x7fd3000b0410 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 msgr2=0x7fd3000b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd3000b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 >> v1:172.21.15.112:6789/0 conn(0x7fd3000af970 legacy=0x7fd3000b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 monclient: found mon.noname-a-legacy 2024-07-27T14:41:24.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.87:6789/0 2024-07-27T14:41:24.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3000b2390 con 0x7fd3000b0410 2024-07-27T14:41:24.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f8a3e10e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a300315b0 0x7f8a300c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd305466640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd3086cc640 5 monclient: authenticate success, global_id 4317 2024-07-27T14:41:24.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.713+0000 7fd3086cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 1 -- 172.21.15.87:0/500459218 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4039070 con 0x7f2abc0735e0 2024-07-27T14:41:24.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: dump: 2024-07-27T14:41:24.729 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 1 -- 172.21.15.87:0/500459218 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab402cbc0 con 0x7f2abc0735e0 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 1 -- 172.21.15.87:0/500459218 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4034440 con 0x7f2abc0735e0 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.714+0000 7f2aba7fc640 10 monclient: dump: 2024-07-27T14:41:24.734 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.735 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f2ac1b45640 5 monclient: authenticate success, global_id 4295 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f2ac1b45640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f2ac1b45640 4 set_mon_vals no callback set 2024-07-27T14:41:24.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42784/0 (socket says 172.21.15.87:42784) 2024-07-27T14:41:24.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 1 -- 172.21.15.87:0/4290895584 learned_addr learned my addr 172.21.15.87:0/4290895584 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 10 monclient(hunting): get_auth_request con 0x7f8a30006c00 auth_method 0 2024-07-27T14:41:24.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.715+0000 7f8a3d90d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f2ac1b45640 10 monclient: shutdown 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f2ac1b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f2ac1b45640 1 -- 172.21.15.87:0/500459218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 msgr2=0x7f2abc0ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/500459218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 0x7f2abc0ada40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009d90 tx=0x7f2ab402be10 comp rx=0 tx=0).stop 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8325a82640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8325a82640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8325a82640 10 monclient: monmap: 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.717418+0000 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.717418+0000 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.741 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.742 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.742 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8a3d90d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8a3d90d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7f8a3d90d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1560156229 0 0) 0x7fd2f80019f0 con 0x7fd3000b0410 2024-07-27T14:41:24.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient: dump: 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] but i'm connected to v1:172.21.15.87:6789/0, reconnecting 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 >> v1:172.21.15.87:6789/0 conn(0x7fd3000b0410 legacy=0x7fd30013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2ec0039e0 0x7fd2ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient(hunting): picked mon.a con 0x7fd2ec0039e0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2ec004800 0x7fd2ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient(hunting): picked mon.b con 0x7fd2ec004800 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd2ec0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient(hunting): picked mon.c con 0x7fd300072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.716+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd300072f50 con 0x7fd2ec0039e0 2024-07-27T14:41:24.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec00bad0 con 0x7fd2ec004800 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec00bcd0 con 0x7fd300072b70 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd305466640 10 monclient(hunting): _finish_auth 0 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 -- 172.21.15.87:0/500459218 shutdown_connections 2024-07-27T14:41:24.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/500459218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0735e0 0x7f2abc0ada40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/500459218 >> v2:172.21.15.112:3300/0 conn(0x7f2abc0abed0 0x7f2abc0ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7fd3086cc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 -- 172.21.15.87:0/500459218 >> 172.21.15.87:0/500459218 conn(0x7f2abc0b4af0 msgr2=0x7f2abc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 -- 172.21.15.87:0/500459218 shutdown_connections 2024-07-27T14:41:24.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f2ac1b45640 1 -- 172.21.15.87:0/500459218 wait complete. 2024-07-27T14:41:24.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f8325a82640 1 Processor -- start 2024-07-27T14:41:24.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f8325a82640 1 -- start start 2024-07-27T14:41:24.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.717+0000 7f8325a82640 10 monclient: init 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f2ac1b45640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f2ac1b45640 10 monclient: monmap: 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.750 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f2ac1b45640 1 librados: starting msgr at 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f2ac1b45640 1 librados: starting objecter 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8320072b70 0x7f83200b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-c con 0x7f8320072b70 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83200af970 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83200b0410 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83200aff60 con 0x7f83200b0410 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83200b3060 con 0x7f83200af970 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f8320072f50 con 0x7f8320072b70 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f8325a82640 10 monclient(hunting): authenticate will time out at 1224.744597s 2024-07-27T14:41:24.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.718+0000 7f2ac1b45640 1 Processor -- start 2024-07-27T14:41:24.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f8a3e10e640 10 monclient(hunting): get_auth_request con 0x7f8a300315b0 auth_method 0 2024-07-27T14:41:24.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f8a3e10e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f8a3e10e640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f8a3e10e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 -- start start 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 librados: setting wanted keys 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 librados: calling monclient init 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient: init 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2abc0735e0 0x7f2abc141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-a con 0x7f2abc0735e0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2abc0abed0 0x7f2abc139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-b con 0x7f2abc0abed0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): picked mon.noname-c con 0x7f2abc0adf80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0ac640 con 0x7f2abc0735e0 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0ac4c0 con 0x7f2abc0abed0 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc1518c0 con 0x7f2abc0adf80 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.719+0000 7f2ac1b45640 10 monclient(hunting): authenticate will time out at 1224.745742s 2024-07-27T14:41:24.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.721+0000 7f8a3e10e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.721+0000 7f8a3e10e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.721+0000 7f8a3e10e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 -- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a300315b0 msgr2=0x7f8a300c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a300315b0 0x7f8a300c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 -- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a3002be70 msgr2=0x7f8a300c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a3002be70 0x7f8a300c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 monclient: found mon.noname-a 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.725+0000 7f8a3d90d640 1 -- 172.21.15.87:0/4290895584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3002a270 con 0x7f8a30006c00 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.727+0000 7fd306c69640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd2ec0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.727+0000 7fd306c69640 10 monclient(hunting): get_auth_request con 0x7fd300072b70 auth_method 0 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.727+0000 7fd306c69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.727+0000 7fd306c69640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.727+0000 7fd306c69640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.728+0000 7f8a3d90d640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400c440 tx=0x7f8a3400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.728+0000 7f8a2e7fc640 10 client.?.objecter ms_handle_connect 0x7f8a30006c00 2024-07-27T14:41:24.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.728+0000 7f8a2e7fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.729+0000 7fd306c69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.729+0000 7fd306c69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.729+0000 7fd306c69640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T14:41:24.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 10 monclient(hunting): handle_auth_done global_id 4317 payload 293 2024-07-27T14:41:24.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 -- 172.21.15.87:0/2682098729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2ec004800 msgr2=0x7fd2ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2ec004800 0x7fd2ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 -- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2ec0039e0 msgr2=0x7fd2ec003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2ec0039e0 0x7fd2ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 monclient: found mon.c 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 1 -- 172.21.15.87:0/2682098729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec003520 con 0x7fd300072b70 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.731+0000 7fd306c69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39290/0 (socket says 172.21.15.87:39290) 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 1 -- 172.21.15.87:0/1850951545 learned_addr learned my addr 172.21.15.87:0/1850951545 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 10 monclient(hunting): get_auth_request con 0x7f2abc0adf80 auth_method 0 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.732+0000 7f2abbfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.733+0000 7f2abbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.733+0000 7f2abbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.733+0000 7f2abbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 -- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2abc0abed0 msgr2=0x7f2abc139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2abc0abed0 0x7f2abc139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 -- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2abc0735e0 msgr2=0x7f2abc141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2abc0735e0 0x7f2abc141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.736+0000 7f2abbfff640 1 -- 172.21.15.87:0/1850951545 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc145dd0 con 0x7f2abc0adf80 2024-07-27T14:41:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7f831ffff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f83200b0410 0x7f832013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45278/0 (socket says 172.21.15.87:45278) 2024-07-27T14:41:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7f831ffff640 1 -- 172.21.15.87:0/3552690881 learned_addr learned my addr 172.21.15.87:0/3552690881 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7fd306c69640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd2ec0051e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800a8a0 tx=0x7fd2ec005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cdcbb080b83ea0a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7f2abbfff640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c390 tx=0x7f2aac00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7f2aa3fff640 10 client.?.objecter ms_handle_connect 0x7f2abc0adf80 2024-07-27T14:41:24.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.740+0000 7f2aa3fff640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34017070 con 0x7f8a30006c00 2024-07-27T14:41:24.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: dump: 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 5 monclient: authenticate success, global_id 4320 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 10 monclient: _renew_subs 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3002a4e0 con 0x7f8a30006c00 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 10 monclient: _renew_subs 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a30028f00 con 0x7f8a30006c00 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a3e90f640 1 librados: init done 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a340035f0 con 0x7f8a30006c00 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a340039e0 con 0x7f8a30006c00 2024-07-27T14:41:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2e7fc640 10 monclient: dump: 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.741+0000 7f8a2ffff640 4 set_mon_vals no callback set 2024-07-27T14:41:24.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.742+0000 7f8a3e90f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.742+0000 7f8a3e90f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.742+0000 7f8a3e90f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.742+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a20005080 con 0x7f8a30006c00 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f800a470 con 0x7fd300072b70 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7fd305466640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7fd305466640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7fd305466640 10 monclient: dump: 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.768 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7fd3086cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293435174 0 0) 0x7f83200aff60 con 0x7f83200b0410 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 10 monclient(hunting): _negotiate 2024-07-27T14:41:24.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 10 monclient(hunting): my global_id is 4323 2024-07-27T14:41:24.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.745+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f830c003520 con 0x7f83200b0410 2024-07-27T14:41:24.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f80072d0 con 0x7fd300072b70 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 1 -- 172.21.15.87:0/2682098729 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80076a0 con 0x7fd300072b70 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd3086cc640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd3086cc640 4 set_mon_vals no callback set 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.748+0000 7fd305466640 10 monclient: dump: 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.749+0000 7fd3086cc640 10 monclient: shutdown 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.749+0000 7fd3086cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.749+0000 7fd3086cc640 1 -- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 msgr2=0x7fd2ec0051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.749+0000 7fd3086cc640 1 --2- 172.21.15.87:0/2682098729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd300072b70 0x7fd2ec0051e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800a8a0 tx=0x7fd2ec005830 comp rx=0 tx=0).stop 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.749+0000 7fd306c69640 1 -- 172.21.15.87:0/2682098729 reap_dead start 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7fd3086cc640 1 -- 172.21.15.87:0/2682098729 shutdown_connections 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7fd3086cc640 1 -- 172.21.15.87:0/2682098729 >> 172.21.15.87:0/2682098729 conn(0x7fd300068150 msgr2=0x7fd3000b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7fd3086cc640 1 -- 172.21.15.87:0/2682098729 shutdown_connections 2024-07-27T14:41:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7fd3086cc640 1 -- 172.21.15.87:0/2682098729 wait complete. 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac017070 con 0x7f2abc0adf80 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: dump: 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac005d00 con 0x7f2abc0adf80 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 5 monclient: authenticate success, global_id 4304 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 10 monclient: _renew_subs 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc140c50 con 0x7f2abc0adf80 2024-07-27T14:41:24.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac007310 con 0x7f2abc0adf80 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2aa3fff640 10 monclient: dump: 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 10 monclient: _renew_subs 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc144f00 con 0x7f2abc0adf80 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ac1b45640 1 librados: init done 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.750+0000 7f2ab9ffb640 4 set_mon_vals no callback set 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 10 monclient: monmap: 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 librados: starting msgr at 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 librados: starting objecter 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 Processor -- start 2024-07-27T14:41:24.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 -- start start 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 librados: setting wanted keys 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 1 librados: calling monclient init 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.751+0000 7fd3086cc640 10 monclient: init 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7f2ac1b45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7f2ac1b45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7f2ac1b45640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:24.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc0ada40 con 0x7f2abc0adf80 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 0x7fd300120a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-b con 0x7fd300072b70 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 0x7fd3001d8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-c con 0x7fd3000af020 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3001d8940 0x7fd30013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): picked mon.noname-a con 0x7fd3001d8940 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001404b0 con 0x7fd3001d8940 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3000b4f30 con 0x7fd300072b70 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fd300140630 con 0x7fd3000af020 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.752+0000 7fd3086cc640 10 monclient(hunting): authenticate will time out at 1224.778905s 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.764+0000 7fd305c67640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 0x7fd3001d8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 0x7fd3001d8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39306/0 (socket says 172.21.15.87:39306) 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 1 -- 172.21.15.87:0/696883687 learned_addr learned my addr 172.21.15.87:0/696883687 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 10 monclient(hunting): get_auth_request con 0x7fd3000af020 auth_method 0 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.771+0000 7fd305c67640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 10 monclient: monmap: 2024-07-27T14:41:24.788 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.788 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.789159+0000 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.789159+0000 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.789 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 1 Processor -- start 2024-07-27T14:41:24.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 1 -- start start 2024-07-27T14:41:24.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.788+0000 7f90dc914640 10 monclient: init 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d8072b70 0x7f90d80b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-b con 0x7f90d8072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f90d80af970 0x7f90d80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-a con 0x7f90d80af970 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90d80b1c80 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90d80aff60 con 0x7f90d80b1c80 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f90d8072f50 con 0x7f90d80af970 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f90d80b3060 con 0x7f90d8072b70 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.789+0000 7f90dc914640 10 monclient(hunting): authenticate will time out at 1224.815981s 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.791+0000 7f5ac8b25640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.791+0000 7f5ac8b25640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.791+0000 7f5ac8b25640 10 monclient: monmap: 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.792294+0000 2024-07-27T14:41:24.793 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.792294+0000 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 Processor -- start 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 -- start start 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient: init 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ac40712b0 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f5ac4071690 0x7f5ac4140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac4071690 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5ac4141300 0x7f5ac414b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac4141300 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac406edc0 con 0x7f5ac40712b0 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac4073100 con 0x7f5ac4071690 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac4071d30 con 0x7f5ac4141300 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.792+0000 7f5ac8b25640 10 monclient(hunting): authenticate will time out at 1224.818966s 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.794+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2aac01a070 con 0x7f2abc0adf80 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.795+0000 7f2aa3fff640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 0x7f2aa806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.795+0000 7f2abb7fe640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 0x7f2aa806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.795+0000 7f2abb7fe640 10 monclient: get_auth_request con 0x7f2aa806d7c0 auth_method 0 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.796+0000 7f2abb7fe640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 0x7f2aa806fc80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009710 tx=0x7f2ab0006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.796+0000 7f2aa3fff640 10 client.4304.objecter ms_handle_connect 0x7f2aa806d7c0 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.864+0000 7f8032694640 10 monclient: get_monmap_and_config 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.864+0000 7f8032694640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.864+0000 7f8032694640 10 monclient: monmap: 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:24.865758+0000 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:24.865758+0000 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:24.865 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:24.866 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:24.866 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:24.866 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:24.866 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.865+0000 7f8032694640 1 Processor -- start 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.865+0000 7f8032694640 1 -- start start 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.865+0000 7f8032694640 10 monclient: init 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f802c0b3630 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f802c0b3a10 0x7f802c143180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): picked mon.noname-a con 0x7f802c0b3a10 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): picked mon.noname-b con 0x7f802c0b3e50 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0abc00 con 0x7f802c0b3630 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f802c139af0 con 0x7f802c0b3a10 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f802c0b30e0 con 0x7f802c0b3e50 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.866+0000 7f8032694640 10 monclient(hunting): authenticate will time out at 1224.892162s 2024-07-27T14:41:24.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.947+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f6be804fd50 con 0x7f6be80ab2e0 2024-07-27T14:41:24.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.947+0000 7f6be4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.947+0000 7f6be4ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:24.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.949+0000 7f5ac259c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5ac40712b0 0x7f5ac413dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36838/0 (socket says 172.21.15.87:36838) 2024-07-27T14:41:24.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.949+0000 7f5ac259c640 1 -- 172.21.15.87:0/2916130840 learned_addr learned my addr 172.21.15.87:0/2916130840 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839672950 0 0) 0x7f5ac406edc0 con 0x7f5ac40712b0 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 10 monclient(hunting): _negotiate 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 10 monclient(hunting): my global_id is 4333 2024-07-27T14:41:24.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.953+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ab0003520 con 0x7f5ac40712b0 2024-07-27T14:41:24.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276371984 0 0) 0x7f83200b3060 con 0x7f83200af970 2024-07-27T14:41:24.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 10 monclient(hunting): _negotiate 2024-07-27T14:41:24.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 10 monclient(hunting): my global_id is 4336 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83200aff60 con 0x7f83200af970 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.959+0000 7f831f7fe640 1 --2- 172.21.15.87:0/3552690881 >> v2:172.21.15.112:3300/0 conn(0x7f8320072b70 0x7f83200b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): get_auth_request con 0x7f8320072b70 auth_method 0 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.960+0000 7f831f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279644285 0 0) 0x7f5ab0003520 con 0x7f5ac40712b0 2024-07-27T14:41:24.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 >> v2:172.21.15.112:3300/0 conn(0x7f5ac4141300 msgr2=0x7f5ac414b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 --2- 172.21.15.87:0/2916130840 >> v2:172.21.15.112:3300/0 conn(0x7f5ac4141300 0x7f5ac414b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 >> v2:172.21.15.87:3300/0 conn(0x7f5ac4071690 msgr2=0x7f5ac4140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 --2- 172.21.15.87:0/2916130840 >> v2:172.21.15.87:3300/0 conn(0x7f5ac4071690 0x7f5ac4140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 monclient: found mon.noname-c-legacy 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.112:6789/0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac40ad730 con 0x7f5ac40712b0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1560156229 0 0) 0x7f5ab4001670 con 0x7f5ac40712b0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-07-27T14:41:24.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: dump: 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] but i'm connected to v1:172.21.15.112:6789/0, reconnecting 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 >> v1:172.21.15.112:6789/0 conn(0x7f5ac40712b0 legacy=0x7f5ac413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 0x7f5ab0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): picked mon.b con 0x7f5ab00039b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ab0004800 0x7f5ab0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): picked mon.a con 0x7f5ab0004800 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00052e0 0x7f5ab000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): picked mon.c con 0x7f5ab00052e0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac4071d30 con 0x7f5ab0004800 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab000c1e0 con 0x7f5ab00039b0 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab000c3e0 con 0x7f5ab00052e0 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:24.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.961+0000 7f5ac1d9b640 10 monclient(hunting): _finish_auth 0 2024-07-27T14:41:24.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.965+0000 7fd306468640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 0x7fd300120a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.965+0000 7fd306468640 10 monclient(hunting): get_auth_request con 0x7fd300072b70 auth_method 0 2024-07-27T14:41:24.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.965+0000 7fd306468640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.965+0000 7fd306468640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.965+0000 7fd306468640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:24.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.966+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.966+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.966+0000 7fd306468640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:24.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 1 -- 172.21.15.87:0/3552690881 >> v1:172.21.15.112:6789/0 conn(0x7f83200af970 legacy=0x7f83200b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 1 -- 172.21.15.87:0/3552690881 >> v1:172.21.15.87:6789/0 conn(0x7f83200b0410 legacy=0x7f832013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 1 monclient: found mon.noname-c 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.968+0000 7f831f7fe640 1 -- 172.21.15.87:0/3552690881 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83200b2390 con 0x7f8320072b70 2024-07-27T14:41:24.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-07-27T14:41:24.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 -- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 msgr2=0x7fd3001d8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 0x7fd3001d8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 -- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3001d8940 msgr2=0x7fd30013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3001d8940 0x7fd30013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 monclient: found mon.noname-b 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.971+0000 7fd306468640 1 -- 172.21.15.87:0/696883687 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd300121aa0 con 0x7fd300072b70 2024-07-27T14:41:24.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.977+0000 7f5ac259c640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 0x7f5ab0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:24.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7f5ac259c640 10 monclient(hunting): get_auth_request con 0x7f5ab00039b0 auth_method 0 2024-07-27T14:41:24.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7f5ac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:24.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7f5ac259c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:24.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7f5ac259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:41:24.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7fd306468640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 0x7fd300120a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc006a90 tx=0x7fd2fc00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7fd2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd300072b70 2024-07-27T14:41:24.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.979+0000 7fd2f6ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:24.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.980+0000 7f831f7fe640 1 --2- 172.21.15.87:0/3552690881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83200b4ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f83100097a0 tx=0x7f831002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f9dc32e9e972a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:24.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.986+0000 7f5ac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:24.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.986+0000 7f5ac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:24.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.986+0000 7f5ac259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310035020 con 0x7f8320072b70 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: dump: 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.988 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8310031830 con 0x7f8320072b70 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f8325a82640 5 monclient: authenticate success, global_id 4339 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f8325a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 1 -- 172.21.15.87:0/3552690881 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310031c00 con 0x7f8320072b70 2024-07-27T14:41:24.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f8325a82640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f8325a82640 4 set_mon_vals no callback set 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.987+0000 7f831e7fc640 10 monclient: dump: 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.990 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc017070 con 0x7fd300072b70 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: dump: 2024-07-27T14:41:24.991 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc00c730 con 0x7fd300072b70 2024-07-27T14:41:24.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00cb20 con 0x7fd300072b70 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd2f6ffd640 10 monclient: dump: 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd304c65640 4 set_mon_vals no callback set 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 10 monclient: shutdown 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 -- 172.21.15.87:0/3552690881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 msgr2=0x7f83200b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 --2- 172.21.15.87:0/3552690881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83200b4ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f83100097a0 tx=0x7f831002b8d0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 -- 172.21.15.87:0/3552690881 shutdown_connections 2024-07-27T14:41:24.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 --2- 172.21.15.87:0/3552690881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83200b4ca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 -- 172.21.15.87:0/3552690881 >> 172.21.15.87:0/3552690881 conn(0x7f8320068150 msgr2=0x7f83200b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 5 monclient: authenticate success, global_id 4342 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 10 monclient: _renew_subs 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd300121d80 con 0x7fd300072b70 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 10 monclient: _renew_subs 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3001d8d80 con 0x7fd300072b70 2024-07-27T14:41:24.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7fd3086cc640 1 librados: init done 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 -- 172.21.15.87:0/3552690881 shutdown_connections 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.988+0000 7f8325a82640 1 -- 172.21.15.87:0/3552690881 wait complete. 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7fd3086cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7fd3086cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7fd3086cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30006ca20 con 0x7fd300072b70 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7f8325a82640 10 monclient: build_initial_monmap 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.989+0000 7f8325a82640 10 monclient: monmap: 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:24.997 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:24.998 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:24.998 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:24.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 10 monclient(hunting): handle_auth_done global_id 4333 payload 293 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 -- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00052e0 msgr2=0x7f5ab000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00052e0 0x7f5ab000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 -- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ab0004800 msgr2=0x7f5ab0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ab0004800 0x7f5ab0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 10 monclient: _finish_hunting 0 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac2d9d640 1 -- 172.21.15.87:0/2916130840 reap_dead start 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 monclient: found mon.b 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 1 -- 172.21.15.87:0/2916130840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac40ad730 con 0x7f5ab00039b0 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 10 monclient: _finish_auth 0 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 10 monclient: _check_auth_tickets 2024-07-27T14:41:24.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 librados: starting msgr at 2024-07-27T14:41:25.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 librados: starting objecter 2024-07-27T14:41:25.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 Processor -- start 2024-07-27T14:41:25.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac8b25640 5 monclient: authenticate success, global_id 4333 2024-07-27T14:41:25.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f5ac8b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 -- start start 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 librados: setting wanted keys 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 1 librados: calling monclient init 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.990+0000 7f8325a82640 10 monclient: init 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83201436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-b con 0x7f8320072b70 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83200af970 0x7f8320141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-c con 0x7f83200af970 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): picked mon.noname-a con 0x7f83200b0410 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:25.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8320140880 con 0x7f83200b0410 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8320140700 con 0x7f8320072b70 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f832014e520 con 0x7f83200af970 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.991+0000 7f8325a82640 10 monclient(hunting): authenticate will time out at 1225.017955s 2024-07-27T14:41:25.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.993+0000 7f5ac259c640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 0x7f5ab0003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c260 tx=0x7f5ab0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=751b30cec3d7c934 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.998+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f2aac056980 con 0x7f2abc0adf80 2024-07-27T14:41:25.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.998+0000 7f2aa3fff640 10 client.4304.objecter ms_dispatch 0x7f2abc000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.998+0000 7f2aa3fff640 3 client.4304.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.998+0000 7f2aa3fff640 3 client.4304.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:24.998+0000 7f2aa3fff640 20 client.4304.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.005+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a3401a070 con 0x7f8a30006c00 2024-07-27T14:41:25.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.006+0000 7f8a2e7fc640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 0x7f8a1c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.006+0000 7f8a3d10c640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 0x7f8a1c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.006+0000 7f8a3d10c640 10 monclient: get_auth_request con 0x7f8a1c06d7c0 auth_method 0 2024-07-27T14:41:25.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.006+0000 7f8a2e7fc640 10 client.4320.objecter ms_handle_connect 0x7f8a1c06d7c0 2024-07-27T14:41:25.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.006+0000 7f8a3d10c640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 0x7f8a1c06fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a28036000 tx=0x7f8a28030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.045+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f8a34056970 con 0x7f8a30006c00 2024-07-27T14:41:25.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.045+0000 7f8a2e7fc640 10 client.4320.objecter ms_dispatch 0x7f8a38000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.045+0000 7f8a2e7fc640 3 client.4320.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.045+0000 7f8a2e7fc640 3 client.4320.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.046+0000 7f8a2e7fc640 20 client.4320.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.046+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f8a34021020 con 0x7f8a30006c00 2024-07-27T14:41:25.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.046+0000 7f8a2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.046+0000 7f8a2e7fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.048+0000 7f90d5d9b640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f90d80af970 0x7f90d80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f90d80af970 0x7f90d80b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42822/0 (socket says 172.21.15.87:42822) 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 1 -- 172.21.15.87:0/1380466785 learned_addr learned my addr 172.21.15.87:0/1380466785 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 10 monclient(hunting): get_auth_request con 0x7f90d80af970 auth_method 0 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.049+0000 7f90d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.051+0000 7f90d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.051+0000 7f90d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.051+0000 7f90d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.053+0000 7f90d5d9b640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.053+0000 7f90d5d9b640 1 -- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d8072b70 msgr2=0x7f90d80b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.053+0000 7f90d5d9b640 1 --2- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d8072b70 0x7f90d80b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.053+0000 7f90d5d9b640 1 -- 172.21.15.87:0/1380466785 >> v1:172.21.15.87:6789/0 conn(0x7f90d80b1c80 legacy=0x7f90d813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.053+0000 7f90d5d9b640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f90d5d9b640 1 monclient: found mon.noname-a 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f90d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f90d5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f90d5d9b640 1 -- 172.21.15.87:0/1380466785 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d81404b0 con 0x7f90d80af970 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42856/0 (socket says 172.21.15.87:42856) 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 1 -- 172.21.15.87:0/1494728691 learned_addr learned my addr 172.21.15.87:0/1494728691 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 10 monclient(hunting): get_auth_request con 0x7f83200b0410 auth_method 0 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.054+0000 7f831ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.055+0000 7f831ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.055+0000 7f831ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.055+0000 7f831ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.056+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f2aac021020 con 0x7f2abc0adf80 2024-07-27T14:41:25.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.056+0000 7f2aa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.056+0000 7f2aa3fff640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.058+0000 7f90d5d9b640 1 --2- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d80af970 0x7f90d80b0410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009870 tx=0x7f90cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2768a6f1f1d84ec server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-07-27T14:41:25.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 -- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83200af970 msgr2=0x7f8320141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83200af970 0x7f8320141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 -- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 msgr2=0x7f83201436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83201436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 monclient: found mon.noname-a 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f831ffff640 1 -- 172.21.15.87:0/1494728691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8320142980 con 0x7f83200b0410 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39316/0 (socket says 172.21.15.87:39316) 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 1 -- 172.21.15.87:0/2789123111 learned_addr learned my addr 172.21.15.87:0/2789123111 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 10 monclient(hunting): get_auth_request con 0x7f802c0b3e50 auth_method 0 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.061+0000 7f8030c31640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.063+0000 7f8030c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.063+0000 7f8030c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.063+0000 7f8030c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 1 -- 172.21.15.87:0/2789123111 >> v2:172.21.15.87:3300/0 conn(0x7f802c0b3a10 msgr2=0x7f802c143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 1 --2- 172.21.15.87:0/2789123111 >> v2:172.21.15.87:3300/0 conn(0x7f802c0b3a10 0x7f802c143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 1 -- 172.21.15.87:0/2789123111 >> v1:172.21.15.87:6789/0 conn(0x7f802c0b3630 legacy=0x7f802c0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 1 monclient: found mon.noname-b 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.066+0000 7f8030c31640 1 -- 172.21.15.87:0/2789123111 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c0abe70 con 0x7f802c0b3e50 2024-07-27T14:41:25.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.067+0000 7f831ffff640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f831800cb10 tx=0x7f831800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.067+0000 7f8317fff640 10 client.?.objecter ms_handle_connect 0x7f83200b0410 2024-07-27T14:41:25.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.067+0000 7f8317fff640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:25.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.069+0000 7f8030c31640 1 --2- 172.21.15.87:0/2789123111 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8024009e50 tx=0x7f802402bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96731945e336861a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 1 -- 172.21.15.87:0/1380466785 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc0317c0 con 0x7f90d80af970 2024-07-27T14:41:25.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: dump: 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.076 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 1 -- 172.21.15.87:0/1380466785 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90cc037070 con 0x7f90d80af970 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 1 -- 172.21.15.87:0/1380466785 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc031b80 con 0x7f90d80af970 2024-07-27T14:41:25.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90d559a640 10 monclient: dump: 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.078 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90dc914640 5 monclient: authenticate success, global_id 4326 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90dc914640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:25.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.075+0000 7f90dc914640 4 set_mon_vals no callback set 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 10 monclient: shutdown 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 -- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d80af970 msgr2=0x7f90d80b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 --2- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d80af970 0x7f90d80b0410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009870 tx=0x7f90cc02b860 comp rx=0 tx=0).stop 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 -- 172.21.15.87:0/1380466785 shutdown_connections 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 --2- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d80af970 0x7f90d80b0410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 --2- 172.21.15.87:0/1380466785 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d8072b70 0x7f90d80b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 -- 172.21.15.87:0/1380466785 >> 172.21.15.87:0/1380466785 conn(0x7f90d8068150 msgr2=0x7f90d80b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 -- 172.21.15.87:0/1380466785 shutdown_connections 2024-07-27T14:41:25.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f90dc914640 1 -- 172.21.15.87:0/1380466785 wait complete. 2024-07-27T14:41:25.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 1 -- 172.21.15.87:0/2789123111 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802402c950 con 0x7f802c0b3e50 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: dump: 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.082 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 1 -- 172.21.15.87:0/2789123111 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802402caf0 con 0x7f802c0b3e50 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 1 -- 172.21.15.87:0/2789123111 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80240343f0 con 0x7f802c0b3e50 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f802affd640 10 monclient: dump: 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.083 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f8032694640 5 monclient: authenticate success, global_id 4310 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f8032694640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.076+0000 7f8032694640 4 set_mon_vals no callback set 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 10 monclient: build_initial_monmap 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 10 monclient: monmap: 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:25.085 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.086 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:25.086 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:25.086 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:25.086 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 10 monclient: shutdown 2024-07-27T14:41:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 -- 172.21.15.87:0/2789123111 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 msgr2=0x7f802c1436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 --2- 172.21.15.87:0/2789123111 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8024009e50 tx=0x7f802402bdb0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 -- 172.21.15.87:0/2789123111 shutdown_connections 2024-07-27T14:41:25.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 --2- 172.21.15.87:0/2789123111 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3e50 0x7f802c1436c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 --2- 172.21.15.87:0/2789123111 >> v2:172.21.15.87:3300/0 conn(0x7f802c0b3a10 0x7f802c143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 -- 172.21.15.87:0/2789123111 >> 172.21.15.87:0/2789123111 conn(0x7f802c073120 msgr2=0x7f802c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 1 librados: starting msgr at 2024-07-27T14:41:25.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 1 librados: starting objecter 2024-07-27T14:41:25.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 -- 172.21.15.87:0/2789123111 shutdown_connections 2024-07-27T14:41:25.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f8032694640 1 -- 172.21.15.87:0/2789123111 wait complete. 2024-07-27T14:41:25.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 1 Processor -- start 2024-07-27T14:41:25.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 1 -- start start 2024-07-27T14:41:25.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.077+0000 7f90dc914640 1 librados: setting wanted keys 2024-07-27T14:41:25.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f90dc914640 1 librados: calling monclient init 2024-07-27T14:41:25.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f90dc914640 10 monclient: init 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f8032694640 10 monclient: build_initial_monmap 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f8032694640 10 monclient: monmap: 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f8032694640 1 librados: starting msgr at 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f8032694640 1 librados: starting objecter 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8032694640 1 Processor -- start 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f90dc914640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.078+0000 7f90dc914640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d8072b70 0x7f90d813ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-a con 0x7f90d8072b70 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90d80af970 0x7f90d8145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-b con 0x7f90d80af970 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): picked mon.noname-c con 0x7f90d80b1c80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f90d80b2520 con 0x7f90d8072b70 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f90d80b23a0 con 0x7f90d80af970 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f90d8146340 con 0x7f90d80b1c80 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:25.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f90dc914640 10 monclient(hunting): authenticate will time out at 1225.105090s 2024-07-27T14:41:25.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8032694640 1 -- start start 2024-07-27T14:41:25.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8032694640 1 librados: setting wanted keys 2024-07-27T14:41:25.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8032694640 1 librados: calling monclient init 2024-07-27T14:41:25.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8032694640 10 monclient: init 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8318017070 con 0x7f83200b0410 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 10 monclient: dump: 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.095 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831800ac70 con 0x7f83200b0410 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8325a82640 5 monclient: authenticate success, global_id 4329 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.079+0000 7f8325a82640 10 monclient: _renew_subs 2024-07-27T14:41:25.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8320142c60 con 0x7f83200b0410 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8317fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831800ae60 con 0x7f83200b0410 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8317fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8317fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8317fff640 10 monclient: dump: 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.097 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.098 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): picked mon.noname-c con 0x7f802c0b3630 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f802c0b3a10 0x7f802c137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): picked mon.noname-a con 0x7f802c0b3a10 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f802c0b3e50 0x7f802c138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): picked mon.noname-b con 0x7f802c0b3e50 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:25.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0b47d0 con 0x7f802c0b3a10 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0b4650 con 0x7f802c0b3e50 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f802c14d4d0 con 0x7f802c0b3630 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8032694640 10 monclient(hunting): authenticate will time out at 1225.106481s 2024-07-27T14:41:25.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f90d6d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f831dffb640 4 set_mon_vals no callback set 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 10 monclient: _renew_subs 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8320146c60 con 0x7f83200b0410 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.080+0000 7f8325a82640 1 librados: init done 2024-07-27T14:41:25.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f802b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f802c0b3a10 0x7f802c137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39346/0 (socket says 172.21.15.87:39346) 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 1 -- 172.21.15.87:0/3323892630 learned_addr learned my addr 172.21.15.87:0/3323892630 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 10 monclient(hunting): get_auth_request con 0x7f90d80b1c80 auth_method 0 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.081+0000 7f90d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f8325a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d659c640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d8072b70 0x7f90d813ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f8325a82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f8325a82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83200b4ca0 con 0x7f83200b0410 2024-07-27T14:41:25.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d659c640 10 monclient(hunting): get_auth_request con 0x7f90d8072b70 auth_method 0 2024-07-27T14:41:25.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f90d659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39360/0 (socket says 172.21.15.87:39360) 2024-07-27T14:41:25.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 1 -- 172.21.15.87:0/1421532091 learned_addr learned my addr 172.21.15.87:0/1421532091 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 10 monclient(hunting): get_auth_request con 0x7f802c0b3630 auth_method 0 2024-07-27T14:41:25.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802b7fe640 10 monclient(hunting): get_auth_request con 0x7f802c0b3a10 auth_method 0 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.082+0000 7f802b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.083+0000 7f802bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.083+0000 7f802bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.083+0000 7f802bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 -- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90d80af970 msgr2=0x7f90d8145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90d80af970 0x7f90d8145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 -- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d8072b70 msgr2=0x7f90d813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d8072b70 0x7f90d813ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 monclient: found mon.noname-c 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.087+0000 7f90d6d9d640 1 -- 172.21.15.87:0/3323892630 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d8146d00 con 0x7f90d80b1c80 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.091+0000 7f802b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.091+0000 7f802b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.091+0000 7f802b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 -- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f802c0b3e50 msgr2=0x7f802c138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f802c0b3e50 0x7f802c138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 -- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f802c0b3a10 msgr2=0x7f802c137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f802c0b3a10 0x7f802c137ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 monclient: found mon.noname-c 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.092+0000 7f802bfff640 1 -- 172.21.15.87:0/1421532091 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c13f990 con 0x7f802c0b3630 2024-07-27T14:41:25.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.096+0000 7f90d6d9d640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f90d000b9c0 tx=0x7f90d000be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.096+0000 7f90c6ffd640 10 client.?.objecter ms_handle_connect 0x7f90d80b1c80 2024-07-27T14:41:25.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.096+0000 7f90c6ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:25.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.097+0000 7f802bfff640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c410 tx=0x7f801c00c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.097+0000 7f8028ff9640 10 client.?.objecter ms_handle_connect 0x7f802c0b3630 2024-07-27T14:41:25.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.097+0000 7f8028ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:25.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0019070 con 0x7f90d80b1c80 2024-07-27T14:41:25.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: dump: 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d000cba0 con 0x7f90d80b1c80 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0014440 con 0x7f90d80b1c80 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90c6ffd640 10 monclient: dump: 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.114 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90dc914640 5 monclient: authenticate success, global_id 4313 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90dc914640 10 monclient: _renew_subs 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90dc914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90dc914640 1 -- 172.21.15.87:0/3323892630 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d8146f90 con 0x7f90d80b1c80 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.104+0000 7f90d4d99640 4 set_mon_vals no callback set 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f90dc914640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f90dc914640 10 monclient: _renew_subs 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f90dc914640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f90dc914640 1 -- 172.21.15.87:0/3323892630 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d81473d0 con 0x7f90d80b1c80 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f90dc914640 1 librados: init done 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c017070 con 0x7f802c0b3630 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: dump: 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.117 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801c0035f0 con 0x7f802c0b3630 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f8028ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.105+0000 7f802a7fc640 4 set_mon_vals no callback set 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c0039e0 con 0x7f802c0b3630 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8028ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8028ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8028ff9640 10 monclient: dump: 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.118 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 5 monclient: authenticate success, global_id 4316 2024-07-27T14:41:25.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 10 monclient: _renew_subs 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802c13fc00 con 0x7f802c0b3630 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 10 monclient: _renew_subs 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f802c138940 con 0x7f802c0b3630 2024-07-27T14:41:25.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.106+0000 7f8032694640 1 librados: init done 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f90c4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f90c4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f90c4ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d8140890 con 0x7f90d80b1c80 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f8032694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f8032694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f8032694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.107+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f802c0b8070 con 0x7f802c0b3630 2024-07-27T14:41:25.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.121+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f831801a070 con 0x7f83200b0410 2024-07-27T14:41:25.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.122+0000 7f8317fff640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 0x7f830006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.122+0000 7f831f7fe640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 0x7f830006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.122+0000 7f831f7fe640 10 monclient: get_auth_request con 0x7f830006d7c0 auth_method 0 2024-07-27T14:41:25.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.123+0000 7f8317fff640 10 client.4329.objecter ms_handle_connect 0x7f830006d7c0 2024-07-27T14:41:25.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.123+0000 7f831f7fe640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 0x7f830006fc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8310036000 tx=0x7f8310030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.152+0000 7f6bed78c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:25.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.152+0000 7f6bed78c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:25.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.152+0000 7f6bed78c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.152+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6be80ac480 con 0x7f6be80ab2e0 2024-07-27T14:41:25.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.259+0000 7f6be4ff9640 1 -- 172.21.15.87:0/2703091661 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6bd8051020 con 0x7f6be80ab2e0 2024-07-27T14:41:25.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.259+0000 7f6be4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:25.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.259+0000 7f6be4ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.260 INFO:teuthology.orchestra.run.smithi087.stdout:55834574853 2024-07-27T14:41:25.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.259+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd2fc01a070 con 0x7fd300072b70 2024-07-27T14:41:25.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.260+0000 7fd2f6ffd640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 0x7fd2e406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.260+0000 7fd305c67640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 0x7fd2e406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.260+0000 7fd305c67640 10 monclient: get_auth_request con 0x7fd2e406d7c0 auth_method 0 2024-07-27T14:41:25.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7fd2f6ffd640 10 client.4342.objecter ms_handle_connect 0x7fd2e406d7c0 2024-07-27T14:41:25.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7fd305c67640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 0x7fd2e406fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd3001474b0 tx=0x7fd2f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7f6bed78c640 10 librados: watch_flush enter 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7f6bed78c640 10 librados: watch_flush exit 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7f6bed78c640 20 client.4330.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.261+0000 7f6bed78c640 10 client.4330.objecter successfully canceled tick 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 msgr2=0x7f6bd406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 0x7f6bd406fd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00b240 tx=0x7f6bdc00b3a0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 10 monclient: shutdown 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 msgr2=0x7f6be8085490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 0x7f6be8085490 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8036000 tx=0x7f6bd8004730 comp rx=0 tx=0).stop 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 shutdown_connections 2024-07-27T14:41:25.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bd406d840 0x7f6bd406fd00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6be80b1a80 0x7f6be81f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6be80b0550 0x7f6be80859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 --2- 172.21.15.87:0/2703091661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6be80ab2e0 0x7f6be8085490 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 >> 172.21.15.87:0/2703091661 conn(0x7f6be80b4c30 msgr2=0x7f6be81e07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 shutdown_connections 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 -- 172.21.15.87:0/2703091661 wait complete. 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.262+0000 7f6bed78c640 1 librados: shutdown 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.264+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7fd2fc057750 con 0x7fd300072b70 2024-07-27T14:41:25.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.264+0000 7fd2f6ffd640 10 client.4342.objecter ms_dispatch 0x7fd300000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.264+0000 7fd2f6ffd640 3 client.4342.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.264+0000 7fd2f6ffd640 3 client.4342.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.264+0000 7fd2f6ffd640 20 client.4342.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.274+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fd30006ca20 con 0x7fd300072b70 2024-07-27T14:41:25.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.274+0000 7fd2f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.274+0000 7fd2f6ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4007d70 con 0x7f5ab00039b0 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: dump: 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.278 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab4030070 con 0x7f5ab00039b0 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 1 -- 172.21.15.87:0/2916130840 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab402cd90 con 0x7f5ab00039b0 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac1d9b640 10 monclient: dump: 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.279 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac8b25640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:25.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.276+0000 7f5ac8b25640 4 set_mon_vals no callback set 2024-07-27T14:41:25.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-07-27T14:41:25.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.277+0000 7f5ac8b25640 10 monclient: shutdown 2024-07-27T14:41:25.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.277+0000 7f5ac8b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.277+0000 7f5ac8b25640 1 -- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 msgr2=0x7f5ab0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 0x7f5ab0003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c260 tx=0x7f5ab0005830 comp rx=0 tx=0).stop 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 -- 172.21.15.87:0/2916130840 shutdown_connections 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/2916130840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ab00039b0 0x7f5ab0003dd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 -- 172.21.15.87:0/2916130840 >> 172.21.15.87:0/2916130840 conn(0x7f5ac40b6be0 msgr2=0x7f5ac40b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 -- 172.21.15.87:0/2916130840 shutdown_connections 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.278+0000 7f5ac8b25640 1 -- 172.21.15.87:0/2916130840 wait complete. 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.279+0000 7f5ac8b25640 10 monclient: build_initial_monmap 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.279+0000 7f5ac8b25640 10 monclient: monmap: 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:25.286 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 librados: starting msgr at 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 librados: starting objecter 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 Processor -- start 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 -- start start 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 librados: setting wanted keys 2024-07-27T14:41:25.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 1 librados: calling monclient init 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.280+0000 7f5ac8b25640 10 monclient: init 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00039b0 0x7f5ac4144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab00039b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.281+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40ac780 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ac41d3d30 0x7f5ac40aa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac41d3d30 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac41418f0 con 0x7f5ac41d3d30 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac4141770 con 0x7f5ac40ac780 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac4141a70 con 0x7f5ab00039b0 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5ac8b25640 10 monclient(hunting): authenticate will time out at 1225.308323s 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.282+0000 7f5abbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58670/0 (socket says 172.21.15.87:58670) 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 1 -- 172.21.15.87:0/124324710 learned_addr learned my addr 172.21.15.87:0/124324710 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): get_auth_request con 0x7f5ac40ac780 auth_method 0 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:25.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.283+0000 7f5abbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-07-27T14:41:25.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 -- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00039b0 msgr2=0x7f5ac4144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00039b0 0x7f5ac4144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 -- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ac41d3d30 msgr2=0x7f5ac40aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ac41d3d30 0x7f5ac40aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 10 monclient: _finish_hunting 0 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 monclient: found mon.noname-b 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.286+0000 7f5abbfff640 1 -- 172.21.15.87:0/124324710 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac40ae480 con 0x7f5ac40ac780 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.288+0000 7f5abbfff640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00b500 tx=0x7f5aac00b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.288+0000 7f5abaffd640 10 client.?.objecter ms_handle_connect 0x7f5ac40ac780 2024-07-27T14:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.288+0000 7f5abaffd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:25.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac014070 con 0x7f5ac40ac780 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: dump: 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.293 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aac010870 con 0x7f5ac40ac780 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 5 monclient: authenticate success, global_id 4345 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac010c60 con 0x7f5ac40ac780 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5abaffd640 10 monclient: dump: 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:25.294 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 10 monclient: _renew_subs 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac40acca0 con 0x7f5ac40ac780 2024-07-27T14:41:25.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac159a640 4 set_mon_vals no callback set 2024-07-27T14:41:25.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:25.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 10 monclient: _renew_subs 2024-07-27T14:41:25.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac40ad1a0 con 0x7f5ac40ac780 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.292+0000 7f5ac8b25640 1 librados: init done 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.294+0000 7f5ac8b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.294+0000 7f5ac8b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.294+0000 7f5ac8b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.294+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac4140dc0 con 0x7f5ac40ac780 2024-07-27T14:41:25.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.329+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f801c01a070 con 0x7f802c0b3630 2024-07-27T14:41:25.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.330+0000 7f8028ff9640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 0x7f801806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.330+0000 7f802b7fe640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 0x7f801806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.330+0000 7f802b7fe640 10 monclient: get_auth_request con 0x7f801806d810 auth_method 0 2024-07-27T14:41:25.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.331+0000 7f8028ff9640 10 client.4316.objecter ms_handle_connect 0x7f801806d810 2024-07-27T14:41:25.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.331+0000 7f802b7fe640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 0x7f801806fcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f802c135f00 tx=0x7f8020009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.338+0000 7f2ac1b45640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:25.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.338+0000 7f2ac1b45640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:25.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.338+0000 7f2ac1b45640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.338+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2abc075b90 con 0x7f2abc0adf80 2024-07-27T14:41:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.342+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f90d001c070 con 0x7f90d80b1c80 2024-07-27T14:41:25.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.342+0000 7f90c6ffd640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 0x7f90b806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.342+0000 7f90d659c640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 0x7f90b806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.342+0000 7f90d659c640 10 monclient: get_auth_request con 0x7f90b806d810 auth_method 0 2024-07-27T14:41:25.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.343+0000 7f90c6ffd640 10 client.4313.objecter ms_handle_connect 0x7f90b806d810 2024-07-27T14:41:25.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.343+0000 7f90d659c640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 0x7f90b806fcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009890 tx=0x7f90c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.345+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f801c0569b0 con 0x7f802c0b3630 2024-07-27T14:41:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.345+0000 7f8028ff9640 10 client.4316.objecter ms_dispatch 0x7f802c000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.345+0000 7f8028ff9640 3 client.4316.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.345+0000 7f8028ff9640 3 client.4316.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.345+0000 7f8028ff9640 20 client.4316.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.353+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f90d00573c0 con 0x7f90d80b1c80 2024-07-27T14:41:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.353+0000 7f90c6ffd640 10 client.4313.objecter ms_dispatch 0x7f90d8000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.353+0000 7f90c6ffd640 3 client.4313.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.353+0000 7f90c6ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.353+0000 7f90c6ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.383+0000 7f8a3e90f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:25.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.383+0000 7f8a3e90f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:25.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.383+0000 7f8a3e90f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.383+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8a20005640 con 0x7f8a30006c00 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f8318057440 con 0x7f83200b0410 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 10 client.4329.objecter ms_dispatch 0x7f8320000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 3 client.4329.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 3 client.4329.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 20 client.4329.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f83180593a0 con 0x7f83200b0410 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.396+0000 7f8317fff640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.411+0000 7f8a2e7fc640 1 -- 172.21.15.87:0/4290895584 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a34058fd0 con 0x7f8a30006c00 2024-07-27T14:41:25.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.411+0000 7f8a2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:25.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.411+0000 7f8a2e7fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.412 INFO:teuthology.orchestra.run.smithi087.stdout:77309411335 2024-07-27T14:41:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 10 librados: watch_flush enter 2024-07-27T14:41:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 10 librados: watch_flush exit 2024-07-27T14:41:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 20 client.4320.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 10 client.4320.objecter successfully canceled tick 2024-07-27T14:41:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 msgr2=0x7f8a1c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 0x7f8a1c06fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a28036000 tx=0x7f8a28030000 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 10 monclient: shutdown 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 msgr2=0x7f8a30029ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400c440 tx=0x7f8a3400c840 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 shutdown_connections 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a1c06d7c0 0x7f8a1c06fc80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a300315b0 0x7f8a300c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a3002be70 0x7f8a300c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 --2- 172.21.15.87:0/4290895584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a30006c00 0x7f8a30029ca0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 >> 172.21.15.87:0/4290895584 conn(0x7f8a30005a50 msgr2=0x7f8a3002c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 shutdown_connections 2024-07-27T14:41:25.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.413+0000 7f8a3e90f640 1 -- 172.21.15.87:0/4290895584 wait complete. 2024-07-27T14:41:25.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.414+0000 7f8a3e90f640 1 librados: shutdown 2024-07-27T14:41:25.424 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411335 for osd.2 2024-07-27T14:41:25.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.518+0000 7fd3086cc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:25.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.518+0000 7fd3086cc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:25.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.518+0000 7fd3086cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.518+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd3001d8040 con 0x7fd300072b70 2024-07-27T14:41:25.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.543+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5aac017070 con 0x7f5ac40ac780 2024-07-27T14:41:25.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.543+0000 7f5abaffd640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 0x7f5aa406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:25.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.543+0000 7f5ac259c640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 0x7f5aa406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:25.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.543+0000 7f5ac259c640 10 monclient: get_auth_request con 0x7f5aa406d730 auth_method 0 2024-07-27T14:41:25.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.544+0000 7f5ac259c640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 0x7f5aa406fbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4034000 tx=0x7f5ab402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:25.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.544+0000 7f5abaffd640 10 client.4345.objecter ms_handle_connect 0x7f5aa406d730 2024-07-27T14:41:25.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.547+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5266+0+0 (secure 0 0 0) 0x7f5aac057530 con 0x7f5ac40ac780 2024-07-27T14:41:25.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.547+0000 7f5abaffd640 10 client.4345.objecter ms_dispatch 0x7f5ac4000b90 osd_map(24..24 src has 1..24) 2024-07-27T14:41:25.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.547+0000 7f5abaffd640 3 client.4345.objecter handle_osd_map got epochs [24,24] > 0 2024-07-27T14:41:25.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.547+0000 7f5abaffd640 3 client.4345.objecter handle_osd_map decoding full epoch 24 2024-07-27T14:41:25.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.547+0000 7f5abaffd640 20 client.4345.objecter dump_active .. 0 homeless 2024-07-27T14:41:25.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.557+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f5aac059b90 con 0x7f5ac40ac780 2024-07-27T14:41:25.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.557+0000 7f5abaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.557+0000 7f5abaffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.564+0000 7fd2f6ffd640 1 -- 172.21.15.87:0/696883687 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd300121aa0 con 0x7fd300072b70 2024-07-27T14:41:25.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.564+0000 7fd2f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:25.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.564+0000 7fd2f6ffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.564 INFO:teuthology.orchestra.run.smithi087.stdout:64424509448 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 10 librados: watch_flush enter 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 10 librados: watch_flush exit 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 20 client.4342.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 10 client.4342.objecter successfully canceled tick 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 msgr2=0x7fd2e406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 0x7fd2e406fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd3001474b0 tx=0x7fd2f0009040 comp rx=0 tx=0).stop 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 10 monclient: shutdown 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 msgr2=0x7fd300120a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 0x7fd300120a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc006a90 tx=0x7fd2fc00ec50 comp rx=0 tx=0).stop 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 shutdown_connections 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd2e406d7c0 0x7fd2e406fc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3001d8940 0x7fd30013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd3000af020 0x7fd3001d8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 --2- 172.21.15.87:0/696883687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd300072b70 0x7fd300120a80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 >> 172.21.15.87:0/696883687 conn(0x7fd300068150 msgr2=0x7fd300141290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 shutdown_connections 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 -- 172.21.15.87:0/696883687 wait complete. 2024-07-27T14:41:25.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.566+0000 7fd3086cc640 1 librados: shutdown 2024-07-27T14:41:25.578 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.1 2024-07-27T14:41:25.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.615+0000 7f8325a82640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:25.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.615+0000 7f8325a82640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:25.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.615+0000 7f8325a82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:25.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.615+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8320075ac0 con 0x7f83200b0410 2024-07-27T14:41:25.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.621+0000 7f8317fff640 1 -- 172.21.15.87:0/1494728691 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f831805d6a0 con 0x7f83200b0410 2024-07-27T14:41:25.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.621+0000 7f8317fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:25.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.621+0000 7f8317fff640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.621 INFO:teuthology.orchestra.run.smithi087.stdout:55834574854 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 10 librados: watch_flush enter 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 10 librados: watch_flush exit 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 20 client.4329.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 10 client.4329.objecter successfully canceled tick 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 msgr2=0x7f830006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 0x7f830006fc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8310036000 tx=0x7f8310030000 comp rx=0 tx=0).stop 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 10 monclient: shutdown 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 msgr2=0x7f8320134a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f831800cb10 tx=0x7f831800a060 comp rx=0 tx=0).stop 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 shutdown_connections 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f830006d7c0 0x7f830006fc80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83200b0410 0x7f8320134a30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83200af970 0x7f8320141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 --2- 172.21.15.87:0/1494728691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8320072b70 0x7f83201436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 >> 172.21.15.87:0/1494728691 conn(0x7f8320068150 msgr2=0x7f83200b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 shutdown_connections 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 -- 172.21.15.87:0/1494728691 wait complete. 2024-07-27T14:41:25.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.623+0000 7f8325a82640 1 librados: shutdown 2024-07-27T14:41:25.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-07-27T14:41:25.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.702+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f801c021020 con 0x7f802c0b3630 2024-07-27T14:41:25.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.702+0000 7f8028ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.702+0000 7f8028ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.717+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f90d0023020 con 0x7f90d80b1c80 2024-07-27T14:41:25.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.717+0000 7f90c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:25.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.717+0000 7f90c6ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:25.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.732+0000 7f2aa3fff640 1 -- 172.21.15.87:0/1850951545 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2aac05de20 con 0x7f2abc0adf80 2024-07-27T14:41:25.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.732+0000 7f2aa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:25.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.732+0000 7f2aa3fff640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.733 INFO:teuthology.orchestra.run.smithi087.stdout:77309411336 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 10 librados: watch_flush enter 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 10 librados: watch_flush exit 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 20 client.4304.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 10 client.4304.objecter successfully canceled tick 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 msgr2=0x7f2aa806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 0x7f2aa806fc80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009710 tx=0x7f2ab0006c90 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 10 monclient: shutdown 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 msgr2=0x7f2abc139850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c390 tx=0x7f2aac00c850 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 shutdown_connections 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2aa806d7c0 0x7f2aa806fc80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2abc0adf80 0x7f2abc139850 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2abc0abed0 0x7f2abc139310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 --2- 172.21.15.87:0/1850951545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2abc0735e0 0x7f2abc141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 >> 172.21.15.87:0/1850951545 conn(0x7f2abc0b4af0 msgr2=0x7f2abc13e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 shutdown_connections 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.734+0000 7f2ac1b45640 1 -- 172.21.15.87:0/1850951545 wait complete. 2024-07-27T14:41:25.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.735+0000 7f2ac1b45640 1 librados: shutdown 2024-07-27T14:41:25.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411337 got 77309411336 for osd.6 2024-07-27T14:41:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.763+0000 7f5ac8b25640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.763+0000 7f5ac8b25640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.763+0000 7f5ac8b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.763+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5ac4075ac0 con 0x7f5ac40ac780 2024-07-27T14:41:25.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.766+0000 7f5abaffd640 1 -- 172.21.15.87:0/124324710 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5aac059490 con 0x7f5ac40ac780 2024-07-27T14:41:25.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.766+0000 7f5abaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:25.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.766+0000 7f5abaffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.767 INFO:teuthology.orchestra.run.smithi087.stdout:55834574854 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 10 librados: watch_flush enter 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 10 librados: watch_flush exit 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 20 client.4345.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 10 client.4345.objecter successfully canceled tick 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 msgr2=0x7f5aa406fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 0x7f5aa406fbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4034000 tx=0x7f5ab402e000 comp rx=0 tx=0).stop 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 10 monclient: shutdown 2024-07-27T14:41:25.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 msgr2=0x7f5ac41d37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00b500 tx=0x7f5aac00b9c0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 shutdown_connections 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f5aa406d730 0x7f5aa406fbf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ac41d3d30 0x7f5ac40aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5ac40ac780 0x7f5ac41d37f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 --2- 172.21.15.87:0/124324710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5ab00039b0 0x7f5ac4144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 >> 172.21.15.87:0/124324710 conn(0x7f5ac40b6be0 msgr2=0x7f5ac413de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 shutdown_connections 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 -- 172.21.15.87:0/124324710 wait complete. 2024-07-27T14:41:25.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.769+0000 7f5ac8b25640 1 librados: shutdown 2024-07-27T14:41:25.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-07-27T14:41:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.918+0000 7f8032694640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.918+0000 7f8032694640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.918+0000 7f8032694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.918+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f802c13c980 con 0x7f802c0b3630 2024-07-27T14:41:25.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.924+0000 7f8028ff9640 1 -- 172.21.15.87:0/1421532091 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f801c059010 con 0x7f802c0b3630 2024-07-27T14:41:25.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.924+0000 7f8028ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:25.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.924+0000 7f8028ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.925 INFO:teuthology.orchestra.run.smithi087.stdout:64424509448 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 10 librados: watch_flush enter 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 10 librados: watch_flush exit 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 20 client.4316.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 10 client.4316.objecter successfully canceled tick 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 msgr2=0x7f801806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 0x7f801806fcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f802c135f00 tx=0x7f8020009110 comp rx=0 tx=0).stop 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 10 monclient: shutdown 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 msgr2=0x7f802c13ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c410 tx=0x7f801c00c810 comp rx=0 tx=0).stop 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 shutdown_connections 2024-07-27T14:41:25.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f801806d810 0x7f801806fcd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f802c0b3e50 0x7f802c138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f802c0b3a10 0x7f802c137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 --2- 172.21.15.87:0/1421532091 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f802c0b3630 0x7f802c13ebd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 >> 172.21.15.87:0/1421532091 conn(0x7f802c073120 msgr2=0x7f802c0b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 shutdown_connections 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 -- 172.21.15.87:0/1421532091 wait complete. 2024-07-27T14:41:25.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.927+0000 7f8032694640 1 librados: shutdown 2024-07-27T14:41:25.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.0 2024-07-27T14:41:25.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.963+0000 7f90c4ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:25.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.963+0000 7f90c4ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:25.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.963+0000 7f90c4ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:25.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.963+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f90d80b4ca0 con 0x7f90d80b1c80 2024-07-27T14:41:25.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.971+0000 7f90c6ffd640 1 -- 172.21.15.87:0/3323892630 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f90d0059a20 con 0x7f90d80b1c80 2024-07-27T14:41:25.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.971+0000 7f90c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:25.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.971+0000 7f90c6ffd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:64424509448 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 10 librados: watch_flush enter 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 10 librados: watch_flush exit 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 20 client.4313.objecter shutdown clearing up homeless session... 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 10 client.4313.objecter successfully canceled tick 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 msgr2=0x7f90b806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 0x7f90b806fcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009890 tx=0x7f90c8008040 comp rx=0 tx=0).stop 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 10 monclient: shutdown 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 msgr2=0x7f90d81467c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f90d000b9c0 tx=0x7f90d000be80 comp rx=0 tx=0).stop 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 shutdown_connections 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f90b806d810 0x7f90b806fcd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90d80b1c80 0x7f90d81467c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90d80af970 0x7f90d8145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 --2- 172.21.15.87:0/3323892630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90d8072b70 0x7f90d813ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 >> 172.21.15.87:0/3323892630 conn(0x7f90d8068150 msgr2=0x7f90d814cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 shutdown_connections 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 -- 172.21.15.87:0/3323892630 wait complete. 2024-07-27T14:41:25.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:25.974+0000 7f90c4ff9640 1 librados: shutdown 2024-07-27T14:41:25.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.3 2024-07-27T14:41:26.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.390+0000 7eff53689640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.390+0000 7eff53689640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.390+0000 7eff53689640 10 monclient: monmap: 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.391781+0000 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.391781+0000 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.391 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.392 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.392 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.392 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.392 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.392 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.391+0000 7eff53689640 1 Processor -- start 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.391+0000 7eff53689640 1 -- start start 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.391+0000 7eff53689640 10 monclient: init 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-07-27T14:41:26.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7eff4c072b70 0x7eff4c0b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): picked mon.noname-c con 0x7eff4c072b70 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff4c0af970 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff4c0b0410 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c0aff60 con 0x7eff4c0b0410 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c0b3060 con 0x7eff4c0af970 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7eff4c072f50 con 0x7eff4c072b70 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.392+0000 7eff53689640 10 monclient(hunting): authenticate will time out at 1226.418429s 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7eff4c072b70 0x7eff4c0b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51c26640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7eff4c0b0410 0x7eff4c13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45300/0 (socket says 172.21.15.87:45300) 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51c26640 1 -- 172.21.15.87:0/2432156521 learned_addr learned my addr 172.21.15.87:0/2432156521 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:26.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 10 monclient(hunting): get_auth_request con 0x7eff4c072b70 auth_method 0 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.393+0000 7eff51425640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.394+0000 7eff51425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.394+0000 7eff51425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 1 -- 172.21.15.87:0/2432156521 >> v1:172.21.15.112:6789/0 conn(0x7eff4c0af970 legacy=0x7eff4c0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 1 -- 172.21.15.87:0/2432156521 >> v1:172.21.15.87:6789/0 conn(0x7eff4c0b0410 legacy=0x7eff4c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 1 monclient: found mon.noname-c 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.396+0000 7eff51425640 1 -- 172.21.15.87:0/2432156521 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c0b2390 con 0x7eff4c072b70 2024-07-27T14:41:26.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.400+0000 7eff51425640 1 --2- 172.21.15.87:0/2432156521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c072b70 0x7eff4c0b4ca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff480097a0 tx=0x7eff4802b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966e8f8f8f6e53d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 1 -- 172.21.15.87:0/2432156521 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48037070 con 0x7eff4c072b70 2024-07-27T14:41:26.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 10 monclient: dump: 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff53689640 5 monclient: authenticate success, global_id 4348 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.405+0000 7eff53689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 1 -- 172.21.15.87:0/2432156521 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff48030710 con 0x7eff4c072b70 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 1 -- 172.21.15.87:0/2432156521 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48030ae0 con 0x7eff4c072b70 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff43fff640 10 monclient: dump: 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff53689640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.407+0000 7eff53689640 4 set_mon_vals no callback set 2024-07-27T14:41:26.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 10 monclient: shutdown 2024-07-27T14:41:26.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 -- 172.21.15.87:0/2432156521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c072b70 msgr2=0x7eff4c0b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 --2- 172.21.15.87:0/2432156521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c072b70 0x7eff4c0b4ca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff480097a0 tx=0x7eff4802b680 comp rx=0 tx=0).stop 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 -- 172.21.15.87:0/2432156521 shutdown_connections 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 --2- 172.21.15.87:0/2432156521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c072b70 0x7eff4c0b4ca0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 -- 172.21.15.87:0/2432156521 >> 172.21.15.87:0/2432156521 conn(0x7eff4c068150 msgr2=0x7eff4c0b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 -- 172.21.15.87:0/2432156521 shutdown_connections 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.408+0000 7eff53689640 1 -- 172.21.15.87:0/2432156521 wait complete. 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.409+0000 7eff53689640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.409+0000 7eff53689640 10 monclient: monmap: 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.411 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.409+0000 7eff53689640 1 librados: starting msgr at 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.409+0000 7eff53689640 1 librados: starting objecter 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.410+0000 7eff53689640 1 Processor -- start 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.410+0000 7eff53689640 1 -- start start 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.410+0000 7eff53689640 1 librados: setting wanted keys 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.410+0000 7eff53689640 1 librados: calling monclient init 2024-07-27T14:41:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.410+0000 7eff53689640 10 monclient: init 2024-07-27T14:41:26.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:26.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff4c072b70 0x7eff4c0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): picked mon.noname-c con 0x7eff4c072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): picked mon.noname-a con 0x7eff4c0af970 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c0b0410 0x7eff4c0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): picked mon.noname-b con 0x7eff4c0b0410 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c140880 con 0x7eff4c0af970 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c140700 con 0x7eff4c0b0410 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c14e520 con 0x7eff4c072b70 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff53689640 10 monclient(hunting): authenticate will time out at 1226.437281s 2024-07-27T14:41:26.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff50c24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.411+0000 7eff51c26640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c0b0410 0x7eff4c0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42908/0 (socket says 172.21.15.87:42908) 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 1 -- 172.21.15.87:0/263931929 learned_addr learned my addr 172.21.15.87:0/263931929 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): get_auth_request con 0x7eff4c0af970 auth_method 0 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): get_auth_request con 0x7eff4c0b0410 auth_method 0 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff50c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.412+0000 7eff51c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-07-27T14:41:26.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 -- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff4c072b70 msgr2=0x7eff4c0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff4c072b70 0x7eff4c0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 -- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c0b0410 msgr2=0x7eff4c0abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c0b0410 0x7eff4c0abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 monclient: found mon.noname-a 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.415+0000 7eff50c24640 1 -- 172.21.15.87:0/263931929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c147c40 con 0x7eff4c0af970 2024-07-27T14:41:26.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.419+0000 7eff50c24640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00c430 tx=0x7eff3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.419+0000 7eff41ffb640 10 client.?.objecter ms_handle_connect 0x7eff4c0af970 2024-07-27T14:41:26.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.419+0000 7eff41ffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-27T14:41:26.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c011790 con 0x7eff4c0af970 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: dump: 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c017070 con 0x7eff4c0af970 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c011bb0 con 0x7eff4c0af970 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff41ffb640 10 monclient: dump: 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.430 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 5 monclient: authenticate success, global_id 4341 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 10 monclient: _renew_subs 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4c147eb0 con 0x7eff4c0af970 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff437fe640 4 set_mon_vals no callback set 2024-07-27T14:41:26.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 10 monclient: _renew_subs 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4c1472d0 con 0x7eff4c0af970 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.425+0000 7eff53689640 1 librados: init done 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.427+0000 7eff53689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.427+0000 7eff53689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.427+0000 7eff53689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.427+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c0b4ca0 con 0x7eff4c0af970 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.529+0000 7fa5d3213640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.529+0000 7fa5d3213640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.529+0000 7fa5d3213640 10 monclient: monmap: 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.530865+0000 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.530865+0000 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.530 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.531 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.530+0000 7fa5d3213640 1 Processor -- start 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.530+0000 7fa5d3213640 1 -- start start 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.530+0000 7fa5d3213640 10 monclient: init 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5cc072b70 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa5cc0af970 0x7fa5cc0b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-c con 0x7fa5cc0af970 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:26.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-b con 0x7fa5cc0b0410 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5cc0aff60 con 0x7fa5cc072b70 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fa5cc072f50 con 0x7fa5cc0af970 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa5cc0b3060 con 0x7fa5cc0b0410 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d3213640 10 monclient(hunting): authenticate will time out at 1226.557344s 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5d17b0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.531+0000 7fa5cbfff640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa5cc0af970 0x7fa5cc0b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39400/0 (socket says 172.21.15.87:39400) 2024-07-27T14:41:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 1 -- 172.21.15.87:0/1660786638 learned_addr learned my addr 172.21.15.87:0/1660786638 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 10 monclient(hunting): get_auth_request con 0x7fa5cc0b0410 auth_method 0 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5d17b0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5cbfff640 10 monclient(hunting): get_auth_request con 0x7fa5cc0af970 auth_method 0 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5cbfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.532+0000 7fa5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.533+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 1 -- 172.21.15.87:0/1660786638 >> v2:172.21.15.112:3300/0 conn(0x7fa5cc0af970 msgr2=0x7fa5cc0b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 1 --2- 172.21.15.87:0/1660786638 >> v2:172.21.15.112:3300/0 conn(0x7fa5cc0af970 0x7fa5cc0b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 1 -- 172.21.15.87:0/1660786638 >> v1:172.21.15.112:6789/0 conn(0x7fa5cc072b70 legacy=0x7fa5cc0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 1 monclient: found mon.noname-b 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.535+0000 7fa5d17b0640 1 -- 172.21.15.87:0/1660786638 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc0b2390 con 0x7fa5cc0b0410 2024-07-27T14:41:26.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.537+0000 7fa5d17b0640 1 --2- 172.21.15.87:0/1660786638 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009870 tx=0x7fa5b402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1fb73651efacfb server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 1 -- 172.21.15.87:0/1660786638 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4035020 con 0x7fa5cc0b0410 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: dump: 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.542 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 1 -- 172.21.15.87:0/1660786638 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b4031640 con 0x7fa5cc0b0410 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5d3213640 5 monclient: authenticate success, global_id 4319 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5d3213640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5d3213640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5d3213640 4 set_mon_vals no callback set 2024-07-27T14:41:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 1 -- 172.21.15.87:0/1660786638 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b402d040 con 0x7fa5cc0b0410 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.541+0000 7fa5cb7fe640 10 monclient: dump: 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.544 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.545 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 10 monclient: shutdown 2024-07-27T14:41:26.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 -- 172.21.15.87:0/1660786638 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 msgr2=0x7fa5cc13fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 --2- 172.21.15.87:0/1660786638 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009870 tx=0x7fa5b402b8f0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 -- 172.21.15.87:0/1660786638 shutdown_connections 2024-07-27T14:41:26.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 --2- 172.21.15.87:0/1660786638 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13fda0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 --2- 172.21.15.87:0/1660786638 >> v2:172.21.15.112:3300/0 conn(0x7fa5cc0af970 0x7fa5cc0b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 -- 172.21.15.87:0/1660786638 >> 172.21.15.87:0/1660786638 conn(0x7fa5cc068150 msgr2=0x7fa5cc0b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 -- 172.21.15.87:0/1660786638 shutdown_connections 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 1 -- 172.21.15.87:0/1660786638 wait complete. 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.542+0000 7fa5d3213640 10 monclient: monmap: 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.550 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 librados: starting msgr at 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 librados: starting objecter 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 Processor -- start 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 -- start start 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 librados: setting wanted keys 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 1 librados: calling monclient init 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.543+0000 7fa5d3213640 10 monclient: init 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5cc072b70 0x7fa5cc14bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-a con 0x7fa5cc072b70 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5cc0af970 0x7fa5cc13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-b con 0x7fa5cc0af970 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): picked mon.noname-c con 0x7fa5cc0b0410 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc140670 con 0x7fa5cc072b70 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1404f0 con 0x7fa5cc0af970 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa5cc1407f0 con 0x7fa5cc0b0410 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d3213640 10 monclient(hunting): authenticate will time out at 1226.570141s 2024-07-27T14:41:26.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5cbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5cc0af970 0x7fa5cc13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39406/0 (socket says 172.21.15.87:39406) 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 1 -- 172.21.15.87:0/3084602668 learned_addr learned my addr 172.21.15.87:0/3084602668 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 10 monclient(hunting): get_auth_request con 0x7fa5cc0b0410 auth_method 0 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.544+0000 7fa5d17b0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): get_auth_request con 0x7fa5cc0af970 auth_method 0 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.545+0000 7fa5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 -- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5cc0af970 msgr2=0x7fa5cc13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5cc0af970 0x7fa5cc13ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 -- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5cc072b70 msgr2=0x7fa5cc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5cc072b70 0x7fa5cc14bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 monclient: found mon.noname-c 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.548+0000 7fa5d17b0640 1 -- 172.21.15.87:0/3084602668 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc13f750 con 0x7fa5cc0b0410 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.550+0000 7fa5d17b0640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5b402b930 tx=0x7fa5b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.550+0000 7fa5c97fa640 10 client.?.objecter ms_handle_connect 0x7fa5cc0b0410 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.550+0000 7fa5c97fa640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4035040 con 0x7fa5cc0b0410 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: dump: 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b40399f0 con 0x7fa5cc0b0410 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4039dc0 con 0x7fa5cc0b0410 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 5 monclient: authenticate success, global_id 4322 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 10 monclient: _renew_subs 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc0ab170 con 0x7fa5cc0b0410 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5caffd640 4 set_mon_vals no callback set 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5c97fa640 10 monclient: dump: 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.558 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 10 monclient: _renew_subs 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc0ab730 con 0x7fa5cc0b0410 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.554+0000 7fa5d3213640 1 librados: init done 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.555+0000 7fa5d3213640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.555+0000 7fa5d3213640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.555+0000 7fa5d3213640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.555+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5cc0b8ef0 con 0x7fa5cc0b0410 2024-07-27T14:41:26.579 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-27T14:41:26.635 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-07-27T14:41:26.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.681+0000 7fee7e95e640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.681+0000 7fee7e95e640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.681+0000 7fee7e95e640 10 monclient: monmap: 2024-07-27T14:41:26.682 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.682439+0000 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.682439+0000 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 Processor -- start 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 -- start start 2024-07-27T14:41:26.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient: init 2024-07-27T14:41:26.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 0x7fee780b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-b con 0x7fee78072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee780af970 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee780b0410 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee780aff60 con 0x7fee780b0410 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee780b3060 con 0x7fee780af970 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fee78072f50 con 0x7fee78072b70 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.682+0000 7fee7e95e640 10 monclient(hunting): authenticate will time out at 1226.708981s 2024-07-27T14:41:26.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.685+0000 7fee777fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fee780af970 0x7fee780b1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36876/0 (socket says 172.21.15.87:36876) 2024-07-27T14:41:26.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.685+0000 7fee777fe640 1 -- 172.21.15.87:0/124363953 learned_addr learned my addr 172.21.15.87:0/124363953 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:26.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206131400 0 0) 0x7fee780b3060 con 0x7fee780af970 2024-07-27T14:41:26.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 10 monclient(hunting): _negotiate 2024-07-27T14:41:26.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 10 monclient(hunting): my global_id is 4360 2024-07-27T14:41:26.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.687+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee60003520 con 0x7fee780af970 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.690+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2337494335 0 0) 0x7fee60003520 con 0x7fee780af970 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 msgr2=0x7fee780b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 0x7fee780b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 >> v1:172.21.15.87:6789/0 conn(0x7fee780b0410 legacy=0x7fee7813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 monclient: found mon.noname-c-legacy 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.112:6789/0 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee780b2390 con 0x7fee780af970 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1560156229 0 0) 0x7fee6c001670 con 0x7fee780af970 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee7e95e640 5 monclient: authenticate success, global_id 4360 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee7e95e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: dump: 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.693 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] but i'm connected to v1:172.21.15.112:6789/0, reconnecting 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 >> v1:172.21.15.112:6789/0 conn(0x7fee780af970 legacy=0x7fee780b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 0x7fee600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): picked mon.c con 0x7fee78072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee600040f0 0x7fee60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): picked mon.b con 0x7fee600040f0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee60004e60 0x7fee6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): picked mon.a con 0x7fee60004e60 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee78072f50 con 0x7fee60004e60 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee600055e0 con 0x7fee600040f0 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fee6000bf00 con 0x7fee78072b70 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee76ffd640 10 monclient(hunting): _finish_auth 0 2024-07-27T14:41:26.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.691+0000 7fee7e95e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee600040f0 0x7fee60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): get_auth_request con 0x7fee600040f0 auth_method 0 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.694+0000 7fee777fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 293 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 -- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 msgr2=0x7fee600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee78072b70 0x7fee600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 -- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee60004e60 msgr2=0x7fee6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee60004e60 0x7fee6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 monclient: found mon.b 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 1 -- 172.21.15.87:0/124363953 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee60003520 con 0x7fee600040f0 2024-07-27T14:41:26.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.696+0000 7fee777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.697+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7eff3c01c460 con 0x7eff4c0af970 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.697+0000 7eff41ffb640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 0x7eff3006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.697+0000 7eff51425640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 0x7eff3006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.697+0000 7eff51425640 10 monclient: get_auth_request con 0x7eff3006d810 auth_method 0 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.698+0000 7eff41ffb640 10 client.4341.objecter ms_handle_connect 0x7eff3006d810 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.698+0000 7eff51425640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 0x7eff3006fcd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7eff48034000 tx=0x7eff4802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.698+0000 7fee777fe640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee600040f0 0x7fee60004800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee780afd90 tx=0x7fee60005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d410e4c371f59507 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c007c80 con 0x7fee600040f0 2024-07-27T14:41:26.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:26.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: dump: 2024-07-27T14:41:26.702 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6c030070 con 0x7fee600040f0 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee7e95e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee7e95e640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 1 -- 172.21.15.87:0/124363953 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c005d70 con 0x7fee600040f0 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee7e95e640 4 set_mon_vals no callback set 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.702+0000 7fee76ffd640 10 monclient: dump: 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.704 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 10 monclient: shutdown 2024-07-27T14:41:26.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 -- 172.21.15.87:0/124363953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee600040f0 msgr2=0x7fee60004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 --2- 172.21.15.87:0/124363953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee600040f0 0x7fee60004800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee780afd90 tx=0x7fee60005830 comp rx=0 tx=0).stop 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7cefb640 1 -- 172.21.15.87:0/124363953 reap_dead start 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 -- 172.21.15.87:0/124363953 shutdown_connections 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 -- 172.21.15.87:0/124363953 >> 172.21.15.87:0/124363953 conn(0x7fee78068150 msgr2=0x7fee780b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 -- 172.21.15.87:0/124363953 shutdown_connections 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.703+0000 7fee7e95e640 1 -- 172.21.15.87:0/124363953 wait complete. 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.704+0000 7fee7e95e640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.704+0000 7fee7e95e640 10 monclient: monmap: 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.704+0000 7fee7e95e640 1 librados: starting msgr at 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.704+0000 7fee7e95e640 1 librados: starting objecter 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.704+0000 7fee7e95e640 1 Processor -- start 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 1 -- start start 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 1 librados: setting wanted keys 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 1 librados: calling monclient init 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 10 monclient: init 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee600040f0 0x7fee78120810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-a con 0x7fee600040f0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee780ae770 0x7fee781db660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.705+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-c con 0x7fee780ae770 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): picked mon.noname-b con 0x7fee781dbba0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee781404b0 con 0x7fee600040f0 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee780b4f30 con 0x7fee781dbba0 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fee78140630 con 0x7fee780ae770 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7e95e640 10 monclient(hunting): authenticate will time out at 1226.732040s 2024-07-27T14:41:26.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58726/0 (socket says 172.21.15.87:58726) 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 1 -- 172.21.15.87:0/2571694267 learned_addr learned my addr 172.21.15.87:0/2571694267 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 10 monclient(hunting): get_auth_request con 0x7fee781dbba0 auth_method 0 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.706+0000 7fee7cefb640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.707+0000 7fee7cefb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.707+0000 7fee7cefb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.707+0000 7fee7cefb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 -- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee780ae770 msgr2=0x7fee781db660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee780ae770 0x7fee781db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 -- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee600040f0 msgr2=0x7fee78120810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee600040f0 0x7fee78120810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 monclient: found mon.noname-b 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.710+0000 7fee7cefb640 1 -- 172.21.15.87:0/2571694267 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee780aebb0 con 0x7fee781dbba0 2024-07-27T14:41:26.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7eff3c057a80 con 0x7eff4c0af970 2024-07-27T14:41:26.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 10 client.4341.objecter ms_dispatch 0x7eff4c000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:26.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 3 client.4341.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:26.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 3 client.4341.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:26.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 20 client.4341.objecter dump_active .. 0 homeless 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7eff3c05a0e0 con 0x7eff4c0af970 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.711+0000 7eff41ffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.712+0000 7fee7cefb640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee78073800 tx=0x7fee7000c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.712+0000 7fee74ff9640 10 client.?.objecter ms_handle_connect 0x7fee781dbba0 2024-07-27T14:41:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.712+0000 7fee74ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70007110 con 0x7fee781dbba0 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: dump: 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.717 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee700072d0 con 0x7fee781dbba0 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee74ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.715+0000 7fee767fc640 4 set_mon_vals no callback set 2024-07-27T14:41:26.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 5 monclient: authenticate success, global_id 4363 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee700076c0 con 0x7fee781dbba0 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 10 monclient: _renew_subs 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee780abde0 con 0x7fee781dbba0 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee74ff9640 10 monclient: dump: 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.719 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 10 monclient: _renew_subs 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee78121dc0 con 0x7fee781dbba0 2024-07-27T14:41:26.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.716+0000 7fee7e95e640 1 librados: init done 2024-07-27T14:41:26.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.717+0000 7fee7e95e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.717+0000 7fee7e95e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.717+0000 7fee7e95e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:26.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.717+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee780b1c80 con 0x7fee781dbba0 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.743+0000 7f8430694640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.743+0000 7f8430694640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.743+0000 7f8430694640 10 monclient: monmap: 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.744869+0000 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.744869+0000 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.744 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.745 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.745 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.745 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.745 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.744+0000 7f8430694640 1 Processor -- start 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.744+0000 7f8430694640 1 -- start start 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.744+0000 7f8430694640 10 monclient: init 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): picked mon.noname-b con 0x7f84280b3630 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84280b3a10 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8428143890 0x7f842814d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): picked mon.noname-c con 0x7f8428143890 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84280abc00 con 0x7f84280b3a10 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f8428139af0 con 0x7f8428143890 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f84280abe70 con 0x7f84280b3630 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.745+0000 7f8430694640 10 monclient(hunting): authenticate will time out at 1226.771399s 2024-07-27T14:41:26.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-27T14:41:26.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-27T14:41:26.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.820+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa5b4038020 con 0x7fa5cc0b0410 2024-07-27T14:41:26.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.821+0000 7fa5c97fa640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 0x7fa5ac06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.821+0000 7fa5d0faf640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 0x7fa5ac06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.821+0000 7fa5d0faf640 10 monclient: get_auth_request con 0x7fa5ac06d7c0 auth_method 0 2024-07-27T14:41:26.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.822+0000 7fa5c97fa640 10 client.4322.objecter ms_handle_connect 0x7fa5ac06d7c0 2024-07-27T14:41:26.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.822+0000 7fa5d0faf640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 0x7fa5ac06fc80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4000f60 tx=0x7fa5c40079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7fa5b4079070 con 0x7fa5cc0b0410 2024-07-27T14:41:26.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 10 client.4322.objecter ms_dispatch 0x7fa5cc000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:26.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 3 client.4322.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:26.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 3 client.4322.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:26.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 20 client.4322.objecter dump_active .. 0 homeless 2024-07-27T14:41:26.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fa5cc0b8ef0 con 0x7fa5cc0b0410 2024-07-27T14:41:26.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.838+0000 7fa5c97fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:26.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.849+0000 7f842e430640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39424/0 (socket says 172.21.15.87:39424) 2024-07-27T14:41:26.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 1 -- 172.21.15.87:0/1177956192 learned_addr learned my addr 172.21.15.87:0/1177956192 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 10 monclient(hunting): get_auth_request con 0x7f84280b3630 auth_method 0 2024-07-27T14:41:26.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.850+0000 7f842e430640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.851+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.851+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.851+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 1 -- 172.21.15.87:0/1177956192 >> v2:172.21.15.112:3300/0 conn(0x7f8428143890 msgr2=0x7f842814d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 1 --2- 172.21.15.87:0/1177956192 >> v2:172.21.15.112:3300/0 conn(0x7f8428143890 0x7f842814d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 1 -- 172.21.15.87:0/1177956192 >> v1:172.21.15.112:6789/0 conn(0x7f84280b3a10 legacy=0x7f8428143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 1 monclient: found mon.noname-b 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.855+0000 7f842e430640 1 -- 172.21.15.87:0/1177956192 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8428142c60 con 0x7f84280b3630 2024-07-27T14:41:26.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.856+0000 7f67a67cd640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.856+0000 7f67a67cd640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.856+0000 7f67a67cd640 10 monclient: monmap: 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.857165+0000 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.857165+0000 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.857 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 1 Processor -- start 2024-07-27T14:41:26.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 1 -- start start 2024-07-27T14:41:26.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient: init 2024-07-27T14:41:26.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67a00678a0 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f67a00b0010 0x7f67a0142c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-c con 0x7f67a00b0010 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f67a0143140 0x7f67a014d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-a con 0x7f67a0143140 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.857+0000 7f67a67cd640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67a00b0770 con 0x7f67a00678a0 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f67a0067c80 con 0x7f67a0143140 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f67a00b0430 con 0x7f67a00b0010 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a67cd640 10 monclient(hunting): authenticate will time out at 1226.884104s 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.858+0000 7f67a4d6a640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f67a0143140 0x7f67a014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f67a0143140 0x7f67a014d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42942/0 (socket says 172.21.15.87:42942) 2024-07-27T14:41:26.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 1 -- 172.21.15.87:0/3369838945 learned_addr learned my addr 172.21.15.87:0/3369838945 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): get_auth_request con 0x7f67a0143140 auth_method 0 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.859+0000 7f67a4d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f842e430640 1 --2- 172.21.15.87:0/1177956192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=802e02e698cb1927 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 1 -- 172.21.15.87:0/3369838945 >> v2:172.21.15.112:3300/0 conn(0x7f67a00b0010 msgr2=0x7f67a0142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 1 --2- 172.21.15.87:0/3369838945 >> v2:172.21.15.112:3300/0 conn(0x7f67a00b0010 0x7f67a0142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 1 -- 172.21.15.87:0/3369838945 >> v1:172.21.15.112:6789/0 conn(0x7f67a00678a0 legacy=0x7f67a013ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 1 monclient: found mon.noname-a 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.861+0000 7f67a4d6a640 1 -- 172.21.15.87:0/3369838945 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a01435d0 con 0x7f67a0143140 2024-07-27T14:41:26.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 1 -- 172.21.15.87:0/1177956192 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424038070 con 0x7f84280b3630 2024-07-27T14:41:26.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: dump: 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.867 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f8430694640 5 monclient: authenticate success, global_id 4325 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f8430694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 1 -- 172.21.15.87:0/1177956192 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8424031740 con 0x7f84280b3630 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f842d42e640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f8430694640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f8430694640 4 set_mon_vals no callback set 2024-07-27T14:41:26.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.865+0000 7f67a4d6a640 1 --2- 172.21.15.87:0/3369838945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a0143140 0x7f67a014d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67a01471c0 tx=0x7f679802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f33749e19a3a34 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f8430694640 10 monclient: shutdown 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f8430694640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f8430694640 1 -- 172.21.15.87:0/1177956192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 msgr2=0x7f84280b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f842d42e640 1 -- 172.21.15.87:0/1177956192 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424038070 con 0x7f84280b3630 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f8430694640 1 --2- 172.21.15.87:0/1177956192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402b7e0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.866+0000 7f842d42e640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 -- 172.21.15.87:0/1177956192 shutdown_connections 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 --2- 172.21.15.87:0/1177956192 >> v2:172.21.15.112:3300/0 conn(0x7f8428143890 0x7f842814d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 --2- 172.21.15.87:0/1177956192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f84280b29d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 -- 172.21.15.87:0/1177956192 >> 172.21.15.87:0/1177956192 conn(0x7f8428073120 msgr2=0x7f8428067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 -- 172.21.15.87:0/1177956192 shutdown_connections 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.867+0000 7f8430694640 1 -- 172.21.15.87:0/1177956192 wait complete. 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 10 monclient: monmap: 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 librados: starting msgr at 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 librados: starting objecter 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 Processor -- start 2024-07-27T14:41:26.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 -- start start 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 librados: setting wanted keys 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 1 librados: calling monclient init 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.868+0000 7f8430694640 10 monclient: init 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): picked mon.noname-c con 0x7f84280b3630 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84280b3a10 0x7f8428070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): picked mon.noname-a con 0x7f84280b3a10 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428143890 0x7f842806daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): picked mon.noname-b con 0x7f8428143890 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f84280b3090 con 0x7f84280b3a10 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84280b2f10 con 0x7f8428143890 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f84280b3e50 con 0x7f84280b3630 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f8430694640 10 monclient(hunting): authenticate will time out at 1226.895385s 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.869+0000 7f842e430640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39428/0 (socket says 172.21.15.87:39428) 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 1 -- 172.21.15.87:0/975306524 learned_addr learned my addr 172.21.15.87:0/975306524 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 10 monclient(hunting): get_auth_request con 0x7f84280b3630 auth_method 0 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f842e430640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 1 -- 172.21.15.87:0/3369838945 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679802c960 con 0x7f67a0143140 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: dump: 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.875 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 1 -- 172.21.15.87:0/3369838945 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f679802cb00 con 0x7f67a0143140 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 1 -- 172.21.15.87:0/3369838945 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67980343f0 con 0x7f67a0143140 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f679effd640 10 monclient: dump: 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.876 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f67a67cd640 5 monclient: authenticate success, global_id 4344 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f67a67cd640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.870+0000 7f67a67cd640 4 set_mon_vals no callback set 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842e430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842dc2f640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84280b3a10 0x7f8428070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 10 monclient: shutdown 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 -- 172.21.15.87:0/3369838945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a0143140 msgr2=0x7f67a014d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3369838945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a0143140 0x7f67a014d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67a01471c0 tx=0x7f679802bdc0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 -- 172.21.15.87:0/3369838945 shutdown_connections 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3369838945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a0143140 0x7f67a014d450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3369838945 >> v2:172.21.15.112:3300/0 conn(0x7f67a00b0010 0x7f67a0142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 -- 172.21.15.87:0/3369838945 >> 172.21.15.87:0/3369838945 conn(0x7f67a006b890 msgr2=0x7f67a006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842dc2f640 10 monclient(hunting): get_auth_request con 0x7f84280b3a10 auth_method 0 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842dc2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 -- 172.21.15.87:0/3369838945 shutdown_connections 2024-07-27T14:41:26.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f67a67cd640 1 -- 172.21.15.87:0/3369838945 wait complete. 2024-07-27T14:41:26.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842dc2f640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.871+0000 7f842dc2f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.872+0000 7f842dc2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.872+0000 7f842dc2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.872+0000 7f842dc2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.872+0000 7f67a67cd640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.872+0000 7f67a67cd640 10 monclient: monmap: 2024-07-27T14:41:26.881 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.881 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.881 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 librados: starting msgr at 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 librados: starting objecter 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 Processor -- start 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 -- start start 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 librados: setting wanted keys 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 librados: calling monclient init 2024-07-27T14:41:26.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 10 monclient: init 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.873+0000 7f67a67cd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a00678a0 0x7f67a01ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-a con 0x7f67a00678a0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-c con 0x7f67a00b0010 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67a0143140 0x7f67a01e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): picked mon.noname-b con 0x7f67a0143140 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0068520 con 0x7f67a00678a0 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f67a00683a0 con 0x7f67a0143140 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f67a0140680 con 0x7f67a00b0010 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.874+0000 7f67a67cd640 10 monclient(hunting): authenticate will time out at 1226.900206s 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 -- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428143890 msgr2=0x7f842806daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428143890 0x7f842806daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 -- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84280b3a10 msgr2=0x7f8428070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84280b3a10 0x7f8428070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 monclient: found mon.noname-c 2024-07-27T14:41:26.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.875+0000 7f842e430640 1 -- 172.21.15.87:0/975306524 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842813fa80 con 0x7f84280b3630 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f842e430640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f842402b820 tx=0x7f84240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f841effd640 10 client.?.objecter ms_handle_connect 0x7f84280b3630 2024-07-27T14:41:26.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f841effd640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39436/0 (socket says 172.21.15.87:39436) 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 1 -- 172.21.15.87:0/3634424312 learned_addr learned my addr 172.21.15.87:0/3634424312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 10 monclient(hunting): get_auth_request con 0x7f67a00b0010 auth_method 0 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.877+0000 7f679f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.878+0000 7f679f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.878+0000 7f679f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.878+0000 7f679f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679ffff640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a00678a0 0x7f67a01ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 -- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67a0143140 msgr2=0x7f67a01e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67a0143140 0x7f67a01e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 -- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a00678a0 msgr2=0x7f67a01ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a00678a0 0x7f67a01ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 monclient: found mon.noname-c 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.882+0000 7f679f7fe640 1 -- 172.21.15.87:0/3634424312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a0146e80 con 0x7f67a00b0010 2024-07-27T14:41:26.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f679f7fe640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f679400b4a0 tx=0x7f679400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f679cff9640 10 client.?.objecter ms_handle_connect 0x7f67a00b0010 2024-07-27T14:41:26.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f679cff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424042070 con 0x7f84280b3630 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: dump: 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.890 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8424003810 con 0x7f84280b3630 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 5 monclient: authenticate success, global_id 4328 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 10 monclient: _renew_subs 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842813a450 con 0x7f84280b3630 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f842cc2d640 4 set_mon_vals no callback set 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424038040 con 0x7f84280b3630 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f841effd640 10 monclient: dump: 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.892 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 10 monclient: _renew_subs 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f842813f7a0 con 0x7f84280b3630 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.888+0000 7f8430694640 1 librados: init done 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.889+0000 7f8430694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.889+0000 7f8430694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.889+0000 7f8430694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:26.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.889+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84280b29d0 con 0x7f84280b3630 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.906+0000 7f78b3321640 10 monclient: get_monmap_and_config 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.906+0000 7f78b3321640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.906+0000 7f78b3321640 10 monclient: monmap: 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:26.907749+0000 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:26.907749+0000 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:26.907 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:26.908 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:26.908 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.907+0000 7f78b3321640 1 Processor -- start 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.907+0000 7f78b3321640 1 -- start start 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.907+0000 7f78b3321640 10 monclient: init 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f78ac0b3630 0x7f78ac0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac0b3630 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78ac0b3a10 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f78ac143890 0x7f78ac14d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-c con 0x7f78ac143890 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ac0abc00 con 0x7f78ac0b3a10 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac139af0 con 0x7f78ac0b3630 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac0abe70 con 0x7f78ac143890 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b3321640 10 monclient(hunting): authenticate will time out at 1226.934301s 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.908+0000 7f78b10bd640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f78ac0b3630 0x7f78ac0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f78ac0b3630 0x7f78ac0b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42968/0 (socket says 172.21.15.87:42968) 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 1 -- 172.21.15.87:0/3427763988 learned_addr learned my addr 172.21.15.87:0/3427763988 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): get_auth_request con 0x7f78ac0b3630 auth_method 0 2024-07-27T14:41:26.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.909+0000 7f78b10bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 1 -- 172.21.15.87:0/3427763988 >> v2:172.21.15.112:3300/0 conn(0x7f78ac143890 msgr2=0x7f78ac14d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 1 --2- 172.21.15.87:0/3427763988 >> v2:172.21.15.112:3300/0 conn(0x7f78ac143890 0x7f78ac14d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 1 -- 172.21.15.87:0/3427763988 >> v1:172.21.15.112:6789/0 conn(0x7f78ac0b3a10 legacy=0x7f78ac143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 1 monclient: found mon.noname-a 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.911+0000 7f78b10bd640 1 -- 172.21.15.87:0/3427763988 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac142c60 con 0x7f78ac0b3630 2024-07-27T14:41:26.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.914+0000 7f78b10bd640 1 --2- 172.21.15.87:0/3427763988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac0b3630 0x7f78ac0b29d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009870 tx=0x7f78a802b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676b5ecb568cfa01 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.917+0000 7f78a3fff640 1 -- 172.21.15.87:0/3427763988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8031760 con 0x7f78ac0b3630 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.917+0000 7f78a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: dump: 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.918 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 1 -- 172.21.15.87:0/3427763988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a8037070 con 0x7f78ac0b3630 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 1 -- 172.21.15.87:0/3427763988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8031b20 con 0x7f78ac0b3630 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:26.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78a3fff640 10 monclient: dump: 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 5 monclient: authenticate success, global_id 4350 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:26.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 4 set_mon_vals no callback set 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 10 monclient: shutdown 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 1 -- 172.21.15.87:0/3427763988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac0b3630 msgr2=0x7f78ac0b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.918+0000 7f78b3321640 1 --2- 172.21.15.87:0/3427763988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac0b3630 0x7f78ac0b29d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009870 tx=0x7f78a802b800 comp rx=0 tx=0).stop 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 -- 172.21.15.87:0/3427763988 shutdown_connections 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 --2- 172.21.15.87:0/3427763988 >> v2:172.21.15.112:3300/0 conn(0x7f78ac143890 0x7f78ac14d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 --2- 172.21.15.87:0/3427763988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac0b3630 0x7f78ac0b29d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 -- 172.21.15.87:0/3427763988 >> 172.21.15.87:0/3427763988 conn(0x7f78ac073120 msgr2=0x7f78ac067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 -- 172.21.15.87:0/3427763988 shutdown_connections 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 1 -- 172.21.15.87:0/3427763988 wait complete. 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.919+0000 7f78b3321640 10 monclient: build_initial_monmap 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient: monmap: 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:26.924 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 librados: starting msgr at 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 librados: starting objecter 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 Processor -- start 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 -- start start 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 librados: setting wanted keys 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 librados: calling monclient init 2024-07-27T14:41:26.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient: init 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78ac0b3630 0x7f78ac13a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-b con 0x7f78ac0b3630 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f78ac0b3a10 0x7f78ac146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-c con 0x7f78ac0b3a10 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.920+0000 7f78b3321640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac143890 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac0b3090 con 0x7f78ac143890 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac0b2f10 con 0x7f78ac0b3630 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:26.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac0b3e50 con 0x7f78ac0b3a10 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b3321640 10 monclient(hunting): authenticate will time out at 1226.947024s 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42980/0 (socket says 172.21.15.87:42980) 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 1 -- 172.21.15.87:0/11207269 learned_addr learned my addr 172.21.15.87:0/11207269 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 10 monclient(hunting): get_auth_request con 0x7f78ac143890 auth_method 0 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.921+0000 7f78b18be640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.922+0000 7f78b18be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.922+0000 7f78b18be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:26.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.922+0000 7f78b18be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 -- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f78ac0b3a10 msgr2=0x7f78ac146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f78ac0b3a10 0x7f78ac146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 -- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78ac0b3630 msgr2=0x7f78ac13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78ac0b3630 0x7f78ac13a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 10 monclient: _finish_hunting 0 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 monclient: found mon.noname-a 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.925+0000 7f78b18be640 1 -- 172.21.15.87:0/11207269 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac13ba50 con 0x7f78ac143890 2024-07-27T14:41:26.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.927+0000 7f78b18be640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78a400b9e0 tx=0x7f78a400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.927+0000 7f78a1ffb640 10 client.?.objecter ms_handle_connect 0x7f78ac143890 2024-07-27T14:41:26.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.927+0000 7f78a1ffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4019070 con 0x7f78ac143890 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: dump: 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.932 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a400cb20 con 0x7f78ac143890 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:26.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4014440 con 0x7f78ac143890 2024-07-27T14:41:26.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:26.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:26.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a1ffb640 10 monclient: dump: 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:26.936 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78b3321640 5 monclient: authenticate success, global_id 4353 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78b3321640 10 monclient: _renew_subs 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78b3321640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac13be70 con 0x7f78ac143890 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.931+0000 7f78a37fe640 4 set_mon_vals no callback set 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.932+0000 7f78b3321640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.932+0000 7f78b3321640 10 monclient: _renew_subs 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.932+0000 7f78b3321640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.932+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac139040 con 0x7f78ac143890 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.932+0000 7f78b3321640 1 librados: init done 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.933+0000 7f78b3321640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.933+0000 7f78b3321640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.933+0000 7f78b3321640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.933+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac13b850 con 0x7f78ac143890 2024-07-27T14:41:26.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-27T14:41:26.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.954+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f78a401c070 con 0x7f78ac143890 2024-07-27T14:41:26.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.954+0000 7f78a1ffb640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 0x7f789006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.955+0000 7f78b10bd640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 0x7f789006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.955+0000 7f78b10bd640 10 monclient: get_auth_request con 0x7f789006d7c0 auth_method 0 2024-07-27T14:41:26.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.955+0000 7f78a1ffb640 10 client.4353.objecter ms_handle_connect 0x7f789006d7c0 2024-07-27T14:41:26.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.955+0000 7f78b10bd640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 0x7f789006fc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78a8036000 tx=0x7f78a8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.958+0000 7eff53689640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:26.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.958+0000 7eff53689640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:26.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.958+0000 7eff53689640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:26.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.958+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7eff4c0b8d30 con 0x7eff4c0af970 2024-07-27T14:41:26.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.963+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee70007860 con 0x7fee781dbba0 2024-07-27T14:41:26.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.964+0000 7fee74ff9640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 0x7fee6406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:26.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.964+0000 7fee77fff640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 0x7fee6406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:26.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.964+0000 7fee77fff640 10 monclient: get_auth_request con 0x7fee6406d7f0 auth_method 0 2024-07-27T14:41:26.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.965+0000 7fee77fff640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 0x7fee6406fcb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee680096b0 tx=0x7fee68006ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:26.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.965+0000 7fee74ff9640 10 client.4363.objecter ms_handle_connect 0x7fee6406d7f0 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.967+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7fee70056c40 con 0x7fee781dbba0 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.967+0000 7fee74ff9640 10 client.4363.objecter ms_dispatch 0x7fee78000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.967+0000 7fee74ff9640 3 client.4363.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.967+0000 7fee74ff9640 3 client.4363.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7fee74ff9640 20 client.4363.objecter dump_active .. 0 homeless 2024-07-27T14:41:26.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f78a40575e0 con 0x7f78ac143890 2024-07-27T14:41:26.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7f78a1ffb640 10 client.4353.objecter ms_dispatch 0x7f78ac000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:26.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7f78a1ffb640 3 client.4353.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:26.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7f78a1ffb640 3 client.4353.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:26.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.968+0000 7f78a1ffb640 20 client.4353.objecter dump_active .. 0 homeless 2024-07-27T14:41:26.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.979+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fee70088e20 con 0x7fee781dbba0 2024-07-27T14:41:26.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.979+0000 7fee74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:26.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:26.979+0000 7fee74ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:26.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-27T14:41:27.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.086+0000 7fd83bafa640 10 monclient: get_monmap_and_config 2024-07-27T14:41:27.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.086+0000 7fd83bafa640 10 monclient: build_initial_monmap 2024-07-27T14:41:27.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.086+0000 7fd83bafa640 10 monclient: monmap: 2024-07-27T14:41:27.087 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:27.087921+0000 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:27.087921+0000 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:27.088 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:27.089 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.087+0000 7fd83bafa640 1 Processor -- start 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.087+0000 7fd83bafa640 1 -- start start 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.087+0000 7fd83bafa640 10 monclient: init 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 0x7fd8340b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-b con 0x7fd8340b3630 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8340b3a10 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:27.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd834143890 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8340abc00 con 0x7fd834143890 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8340abe70 con 0x7fd8340b3a10 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fd834139af0 con 0x7fd8340b3630 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.088+0000 7fd83bafa640 10 monclient(hunting): authenticate will time out at 1227.114684s 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.089+0000 7fd839095640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd8340b3a10 0x7fd834143180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:36912/0 (socket says 172.21.15.87:36912) 2024-07-27T14:41:27.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.089+0000 7fd839095640 1 -- 172.21.15.87:0/1330341440 learned_addr learned my addr 172.21.15.87:0/1330341440 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:27.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.094+0000 7f606d718640 10 monclient: get_monmap_and_config 2024-07-27T14:41:27.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.094+0000 7f606d718640 10 monclient: build_initial_monmap 2024-07-27T14:41:27.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.094+0000 7f606d718640 10 monclient: monmap: 2024-07-27T14:41:27.094 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:27.095369+0000 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:27.095369+0000 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:27.095 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 Processor -- start 2024-07-27T14:41:27.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 -- start start 2024-07-27T14:41:27.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient: init 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6068072b70 0x7f60680b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): picked mon.noname-c con 0x7f6068072b70 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f60680af970 0x7f60680b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): picked mon.noname-a con 0x7f60680af970 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60680b1c80 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60680aff60 con 0x7f60680b1c80 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f60680b3060 con 0x7f60680af970 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f60681404b0 con 0x7f6068072b70 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.095+0000 7f606d718640 10 monclient(hunting): authenticate will time out at 1227.121923s 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6068072b70 0x7f60680b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6068072b70 0x7f60680b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58790/0 (socket says 172.21.15.87:58790) 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 1 -- 172.21.15.87:0/3162601153 learned_addr learned my addr 172.21.15.87:0/3162601153 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 10 monclient(hunting): get_auth_request con 0x7f6068072b70 auth_method 0 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.096+0000 7f6066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.097+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.097+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:27.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.097+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-07-27T14:41:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 1 -- 172.21.15.87:0/3162601153 >> v2:172.21.15.87:3300/0 conn(0x7f60680af970 msgr2=0x7f60680b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 1 --2- 172.21.15.87:0/3162601153 >> v2:172.21.15.87:3300/0 conn(0x7f60680af970 0x7f60680b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 1 -- 172.21.15.87:0/3162601153 >> v1:172.21.15.87:6789/0 conn(0x7f60680b1c80 legacy=0x7f606813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 10 monclient: _finish_hunting 0 2024-07-27T14:41:27.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 1 monclient: found mon.noname-c 2024-07-27T14:41:27.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:27.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.098+0000 7f6066ffd640 1 -- 172.21.15.87:0/3162601153 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6068140660 con 0x7f6068072b70 2024-07-27T14:41:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.100+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8424045070 con 0x7f84280b3630 2024-07-27T14:41:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.100+0000 7f841effd640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 0x7f840c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.100+0000 7f842dc2f640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 0x7f840c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.100+0000 7f842dc2f640 10 monclient: get_auth_request con 0x7f840c06d810 auth_method 0 2024-07-27T14:41:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.100+0000 7f6066ffd640 1 --2- 172.21.15.87:0/3162601153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6068072b70 0x7f60680b4ca0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6058009870 tx=0x7f605802b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3279ca98ea0ca74 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.101+0000 7f842dc2f640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 0x7f840c06fcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8428140a90 tx=0x7f8418009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.101+0000 7f841effd640 10 client.4328.objecter ms_handle_connect 0x7f840c06d810 2024-07-27T14:41:27.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 1 -- 172.21.15.87:0/3162601153 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60580317d0 con 0x7f6068072b70 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: dump: 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.105 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 1 -- 172.21.15.87:0/3162601153 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6058037070 con 0x7f6068072b70 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 1 -- 172.21.15.87:0/3162601153 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058031b90 con 0x7f6068072b70 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:27.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f6065ffb640 10 monclient: dump: 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.107 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f606d718640 5 monclient: authenticate success, global_id 4366 2024-07-27T14:41:27.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f606d718640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:27.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.104+0000 7f606d718640 4 set_mon_vals no callback set 2024-07-27T14:41:27.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 10 monclient: shutdown 2024-07-27T14:41:27.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 -- 172.21.15.87:0/3162601153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6068072b70 msgr2=0x7f60680b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 --2- 172.21.15.87:0/3162601153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6068072b70 0x7f60680b4ca0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6058009870 tx=0x7f605802b870 comp rx=0 tx=0).stop 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 -- 172.21.15.87:0/3162601153 shutdown_connections 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 --2- 172.21.15.87:0/3162601153 >> v2:172.21.15.87:3300/0 conn(0x7f60680af970 0x7f60680b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 --2- 172.21.15.87:0/3162601153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6068072b70 0x7f60680b4ca0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 -- 172.21.15.87:0/3162601153 >> 172.21.15.87:0/3162601153 conn(0x7f6068068150 msgr2=0x7f60680b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 -- 172.21.15.87:0/3162601153 shutdown_connections 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.105+0000 7f606d718640 1 -- 172.21.15.87:0/3162601153 wait complete. 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 10 monclient: build_initial_monmap 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 10 monclient: monmap: 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:27.111 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 librados: starting msgr at 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 librados: starting objecter 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 Processor -- start 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 -- start start 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 librados: setting wanted keys 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 1 librados: calling monclient init 2024-07-27T14:41:27.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.106+0000 7f606d718640 10 monclient: init 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6068072b70 0x7f606813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): picked mon.noname-c con 0x7f6068072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60680af970 0x7f606814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): picked mon.noname-a con 0x7f60680af970 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): picked mon.noname-b con 0x7f60680b1c80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f60680b2480 con 0x7f60680af970 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f60680b2300 con 0x7f60680b1c80 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f606814d510 con 0x7f6068072b70 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.107+0000 7f606d718640 10 monclient(hunting): authenticate will time out at 1227.133502s 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58804/0 (socket says 172.21.15.87:58804) 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 1 -- 172.21.15.87:0/305170311 learned_addr learned my addr 172.21.15.87:0/305170311 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): get_auth_request con 0x7f60680b1c80 auth_method 0 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:27.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.108+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 -- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6068072b70 msgr2=0x7f606813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6068072b70 0x7f606813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 -- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60680af970 msgr2=0x7f606814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60680af970 0x7f606814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 10 monclient: _finish_hunting 0 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 monclient: found mon.noname-b 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.111+0000 7f60677fe640 1 -- 172.21.15.87:0/305170311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606814bd80 con 0x7f60680b1c80 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.114+0000 7f60677fe640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f606000b9c0 tx=0x7f606000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.114+0000 7f604f7fe640 10 client.?.objecter ms_handle_connect 0x7f60680b1c80 2024-07-27T14:41:27.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.114+0000 7f604f7fe640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060019070 con 0x7f60680b1c80 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: dump: 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.120 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: _finish_auth 0 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: _check_auth_tickets 2024-07-27T14:41:27.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f606000cb40 con 0x7f60680b1c80 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f606d718640 5 monclient: authenticate success, global_id 4369 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f606d718640 10 monclient: _renew_subs 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f606d718640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606814c2a0 con 0x7f60680b1c80 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f606d718640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6060014440 con 0x7f60680b1c80 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f60657fa640 4 set_mon_vals no callback set 2024-07-27T14:41:27.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.117+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.118+0000 7f604f7fe640 10 monclient: dump: 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.118+0000 7f606d718640 10 monclient: _renew_subs 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.118+0000 7f606d718640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.118+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f606813cc80 con 0x7f60680b1c80 2024-07-27T14:41:27.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.118+0000 7f606d718640 1 librados: init done 2024-07-27T14:41:27.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.119+0000 7f606d718640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.119+0000 7f606d718640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.119+0000 7f606d718640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.119+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60680b4ca0 con 0x7f60680b1c80 2024-07-27T14:41:27.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.136+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f606001c070 con 0x7f60680b1c80 2024-07-27T14:41:27.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.137+0000 7f604f7fe640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 0x7f605406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.137+0000 7f6066ffd640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 0x7f605406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.137+0000 7f6066ffd640 10 monclient: get_auth_request con 0x7f605406d810 auth_method 0 2024-07-27T14:41:27.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.137+0000 7f6066ffd640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 0x7f605406fcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6058036000 tx=0x7f6058030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.137+0000 7f604f7fe640 10 client.4369.objecter ms_handle_connect 0x7f605406d810 2024-07-27T14:41:27.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.139+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f60600573e0 con 0x7f60680b1c80 2024-07-27T14:41:27.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.139+0000 7f604f7fe640 10 client.4369.objecter ms_dispatch 0x7f6068000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:27.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.139+0000 7f604f7fe640 3 client.4369.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:27.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.139+0000 7f604f7fe640 3 client.4369.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:27.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.139+0000 7f604f7fe640 20 client.4369.objecter dump_active .. 0 homeless 2024-07-27T14:41:27.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f8424079570 con 0x7f84280b3630 2024-07-27T14:41:27.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 10 client.4328.objecter ms_dispatch 0x7f8428000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:27.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 3 client.4328.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:27.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 3 client.4328.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:27.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 20 client.4328.objecter dump_active .. 0 homeless 2024-07-27T14:41:27.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f842813a450 con 0x7f84280b3630 2024-07-27T14:41:27.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.179+0000 7f841effd640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:27.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.183+0000 7fd839896640 1 --2- 172.21.15.87:0/1330341440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 0x7fd8340b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794014070 con 0x7f67a00b0010 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: dump: 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.185 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6794010610 con 0x7f67a00b0010 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 5 monclient: authenticate success, global_id 4331 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 10 monclient: _renew_subs 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a0146960 con 0x7f67a00b0010 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794010a00 con 0x7f67a00b0010 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:27.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679cff9640 10 monclient: dump: 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.188 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f679e7fc640 4 set_mon_vals no callback set 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 10 monclient: _renew_subs 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a0145eb0 con 0x7f67a00b0010 2024-07-27T14:41:27.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.184+0000 7f67a67cd640 1 librados: init done 2024-07-27T14:41:27.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): get_auth_request con 0x7fd8340b3630 auth_method 0 2024-07-27T14:41:27.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:27.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:27.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:27.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7f67a67cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7f67a67cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7f67a67cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a0142c00 con 0x7f67a00b0010 2024-07-27T14:41:27.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.185+0000 7fd839896640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 1 -- 172.21.15.87:0/1330341440 >> v1:172.21.15.112:6789/0 conn(0x7fd8340b3a10 legacy=0x7fd834143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 1 -- 172.21.15.87:0/1330341440 >> v1:172.21.15.87:6789/0 conn(0x7fd834143890 legacy=0x7fd83414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 10 monclient: _finish_hunting 0 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 1 monclient: found mon.noname-b 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.187+0000 7fd839896640 1 -- 172.21.15.87:0/1330341440 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd834142c60 con 0x7fd8340b3630 2024-07-27T14:41:27.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.190+0000 7fd839896640 1 --2- 172.21.15.87:0/1330341440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 0x7fd8340b29d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8300097d0 tx=0x7fd83002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b52b26d976dd0d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.212+0000 7fa5d3213640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:27.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.212+0000 7fa5d3213640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:27.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.212+0000 7fa5d3213640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.212+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa5cc0b4c20 con 0x7fa5cc0b0410 2024-07-27T14:41:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.265+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f78a4023070 con 0x7f78ac143890 2024-07-27T14:41:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.265+0000 7f78a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.265+0000 7f78a1ffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.282+0000 7eff41ffb640 1 -- 172.21.15.87:0/263931929 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff3c0599e0 con 0x7eff4c0af970 2024-07-27T14:41:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.282+0000 7eff41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-27T14:41:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.282+0000 7eff41ffb640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.283 INFO:teuthology.orchestra.run.smithi087.stdout:55834574855 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.285+0000 7eff53689640 10 librados: watch_flush enter 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.285+0000 7eff53689640 10 librados: watch_flush exit 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.285+0000 7eff53689640 20 client.4341.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.285+0000 7eff53689640 10 client.4341.objecter successfully canceled tick 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 msgr2=0x7eff3006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 0x7eff3006fcd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7eff48034000 tx=0x7eff4802f000 comp rx=0 tx=0).stop 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 10 monclient: shutdown 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 msgr2=0x7eff4c0ab5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00c430 tx=0x7eff3c00c8f0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 shutdown_connections 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7eff3006d810 0x7eff3006fcd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff4c0b0410 0x7eff4c0abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff4c0af970 0x7eff4c0ab5f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 --2- 172.21.15.87:0/263931929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff4c072b70 0x7eff4c0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 >> 172.21.15.87:0/263931929 conn(0x7eff4c068150 msgr2=0x7eff4c0b7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 shutdown_connections 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 -- 172.21.15.87:0/263931929 wait complete. 2024-07-27T14:41:27.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.286+0000 7eff53689640 1 librados: shutdown 2024-07-27T14:41:27.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.291+0000 7fee7e95e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:27.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.291+0000 7fee7e95e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:27.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.291+0000 7fee7e95e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.291+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fee781dc000 con 0x7fee781dbba0 2024-07-27T14:41:27.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2024-07-27T14:41:27.302 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.376+0000 7f8430694640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.376+0000 7f8430694640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.376+0000 7f8430694640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.376+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f84280008d0 con 0x7f84280b3630 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.377+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f6060023020 con 0x7f60680b1c80 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.377+0000 7f604f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.377+0000 7f604f7fe640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.431+0000 7fee74ff9640 1 -- 172.21.15.87:0/2571694267 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee7000e280 con 0x7fee781dbba0 2024-07-27T14:41:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.431+0000 7fee74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-27T14:41:27.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.431+0000 7fee74ff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.433 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 10 librados: watch_flush enter 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 10 librados: watch_flush exit 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 20 client.4363.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 10 client.4363.objecter successfully canceled tick 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 msgr2=0x7fee6406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 0x7fee6406fcb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fee680096b0 tx=0x7fee68006ca0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 10 monclient: shutdown 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 msgr2=0x7fee781d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee78073800 tx=0x7fee7000c980 comp rx=0 tx=0).stop 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 shutdown_connections 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fee6406d7f0 0x7fee6406fcb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee781dbba0 0x7fee781d7cb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fee780ae770 0x7fee781db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 --2- 172.21.15.87:0/2571694267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee600040f0 0x7fee78120810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 >> 172.21.15.87:0/2571694267 conn(0x7fee78068150 msgr2=0x7fee780b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 shutdown_connections 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 -- 172.21.15.87:0/2571694267 wait complete. 2024-07-27T14:41:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.434+0000 7fee7e95e640 1 librados: shutdown 2024-07-27T14:41:27.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.1 2024-07-27T14:41:27.451 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.466+0000 7f78b3321640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:27.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.466+0000 7f78b3321640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:27.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.466+0000 7f78b3321640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:27.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.466+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f78ac0b29d0 con 0x7f78ac143890 2024-07-27T14:41:27.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.472+0000 7f78a1ffb640 1 -- 172.21.15.87:0/11207269 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f78a4059c40 con 0x7f78ac143890 2024-07-27T14:41:27.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.472+0000 7f78a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-27T14:41:27.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.472+0000 7f78a1ffb640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.472 INFO:teuthology.orchestra.run.smithi087.stdout:55834574856 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 10 librados: watch_flush enter 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 10 librados: watch_flush exit 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 20 client.4353.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 10 client.4353.objecter successfully canceled tick 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 msgr2=0x7f789006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 0x7f789006fc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78a8036000 tx=0x7f78a8030000 comp rx=0 tx=0).stop 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 10 monclient: shutdown 2024-07-27T14:41:27.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 msgr2=0x7f78ac138b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78a400b9e0 tx=0x7f78a400bea0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 shutdown_connections 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f789006d7c0 0x7f789006fc80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78ac143890 0x7f78ac138b00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f78ac0b3a10 0x7f78ac146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 --2- 172.21.15.87:0/11207269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78ac0b3630 0x7f78ac13a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 >> 172.21.15.87:0/11207269 conn(0x7f78ac073120 msgr2=0x7f78ac0b90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 shutdown_connections 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 -- 172.21.15.87:0/11207269 wait complete. 2024-07-27T14:41:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.474+0000 7f78b3321640 1 librados: shutdown 2024-07-27T14:41:27.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.7 2024-07-27T14:41:27.486 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.502+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6794017070 con 0x7f67a00b0010 2024-07-27T14:41:27.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.503+0000 7f679cff9640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 0x7f678c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.503+0000 7f679ffff640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 0x7f678c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.503+0000 7f679ffff640 10 monclient: get_auth_request con 0x7f678c06d7c0 auth_method 0 2024-07-27T14:41:27.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.503+0000 7f679ffff640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 0x7f678c06fc80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67900015d0 tx=0x7f6790009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.503+0000 7f679cff9640 10 client.4331.objecter ms_handle_connect 0x7f678c06d7c0 2024-07-27T14:41:27.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f6794057060 con 0x7f67a00b0010 2024-07-27T14:41:27.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 10 client.4331.objecter ms_dispatch 0x7f67a0000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 3 client.4331.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 3 client.4331.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 20 client.4331.objecter dump_active .. 0 homeless 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f67940596c0 con 0x7f67a00b0010 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.519+0000 7f679cff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.520+0000 7fa5c97fa640 1 -- 172.21.15.87:0/3084602668 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5b407b6d0 con 0x7fa5cc0b0410 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.520+0000 7fa5c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-27T14:41:27.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.520+0000 7fa5c97fa640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.523 INFO:teuthology.orchestra.run.smithi087.stdout:77309411337 2024-07-27T14:41:27.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 10 librados: watch_flush enter 2024-07-27T14:41:27.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 10 librados: watch_flush exit 2024-07-27T14:41:27.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 20 client.4322.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 10 client.4322.objecter successfully canceled tick 2024-07-27T14:41:27.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 msgr2=0x7fa5ac06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 0x7fa5ac06fc80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4000f60 tx=0x7fa5c40079d0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 10 monclient: shutdown 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 msgr2=0x7fa5cc13f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5b402b930 tx=0x7fa5b40046c0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 shutdown_connections 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa5ac06d7c0 0x7fa5ac06fc80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa5cc0b0410 0x7fa5cc13f210 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5cc0af970 0x7fa5cc13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.522+0000 7fa5d3213640 1 --2- 172.21.15.87:0/3084602668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5cc072b70 0x7fa5cc14bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.523+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 >> 172.21.15.87:0/3084602668 conn(0x7fa5cc068150 msgr2=0x7fa5cc0b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.523+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 shutdown_connections 2024-07-27T14:41:27.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.523+0000 7fa5d3213640 1 -- 172.21.15.87:0/3084602668 wait complete. 2024-07-27T14:41:27.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.523+0000 7fa5d3213640 1 librados: shutdown 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 1 -- 172.21.15.87:0/1330341440 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830039070 con 0x7fd8340b3630 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: dump: 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.531 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: _finish_auth 0 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: _check_auth_tickets 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 1 -- 172.21.15.87:0/1330341440 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8300316f0 con 0x7fd8340b3630 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:27.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 1 -- 172.21.15.87:0/1330341440 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830031ac0 con 0x7fd8340b3630 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd838894640 10 monclient: dump: 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd83bafa640 5 monclient: authenticate success, global_id 4334 2024-07-27T14:41:27.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd83bafa640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:27.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.530+0000 7fd83bafa640 4 set_mon_vals no callback set 2024-07-27T14:41:27.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.531+0000 7fd83bafa640 10 monclient: shutdown 2024-07-27T14:41:27.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.531+0000 7fd83bafa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.531+0000 7fd83bafa640 1 -- 172.21.15.87:0/1330341440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 msgr2=0x7fd8340b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.531+0000 7fd83bafa640 1 --2- 172.21.15.87:0/1330341440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 0x7fd8340b29d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8300097d0 tx=0x7fd83002b790 comp rx=0 tx=0).stop 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.532+0000 7fd83bafa640 1 -- 172.21.15.87:0/1330341440 shutdown_connections 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.532+0000 7fd83bafa640 1 --2- 172.21.15.87:0/1330341440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8340b3630 0x7fd8340b29d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.532+0000 7fd83bafa640 1 -- 172.21.15.87:0/1330341440 >> 172.21.15.87:0/1330341440 conn(0x7fd834073120 msgr2=0x7fd834067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.532+0000 7fd83bafa640 1 -- 172.21.15.87:0/1330341440 shutdown_connections 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.532+0000 7fd83bafa640 1 -- 172.21.15.87:0/1330341440 wait complete. 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 10 monclient: build_initial_monmap 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 10 monclient: monmap: 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:27.536 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 librados: starting msgr at 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 librados: starting objecter 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 Processor -- start 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 -- start start 2024-07-27T14:41:27.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 librados: setting wanted keys 2024-07-27T14:41:27.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 1 librados: calling monclient init 2024-07-27T14:41:27.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.533+0000 7fd83bafa640 10 monclient: init 2024-07-27T14:41:27.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7f841effd640 1 -- 172.21.15.87:0/975306524 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f842407b4d0 con 0x7f84280b3630 2024-07-27T14:41:27.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7f841effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-27T14:41:27.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7f841effd640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.539 INFO:teuthology.orchestra.run.smithi087.stdout:55834574856 2024-07-27T14:41:27.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7fd83bafa640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7fd83bafa640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.534+0000 7fd83bafa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8340b3630 0x7fd8341cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-a con 0x7fd8340b3630 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-b con 0x7fd8340b3a10 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd834143890 0x7fd8340ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): picked mon.noname-c con 0x7fd834143890 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd8340b30d0 con 0x7fd8340b3630 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8340b2f50 con 0x7fd8340b3a10 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:27.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fd8340b3e50 con 0x7fd834143890 2024-07-27T14:41:27.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:27.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd83bafa640 10 monclient(hunting): authenticate will time out at 1227.561368s 2024-07-27T14:41:27.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411337 for osd.2 2024-07-27T14:41:27.542 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.535+0000 7fd839095640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:58806/0 (socket says 172.21.15.87:58806) 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 1 -- 172.21.15.87:0/2338493558 learned_addr learned my addr 172.21.15.87:0/2338493558 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): get_auth_request con 0x7fd8340b3a10 auth_method 0 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:27.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7fd839095640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.536+0000 7f8430694640 10 librados: watch_flush enter 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 10 librados: watch_flush exit 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 20 client.4328.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 10 client.4328.objecter successfully canceled tick 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 msgr2=0x7f840c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 0x7f840c06fcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8428140a90 tx=0x7f8418009040 comp rx=0 tx=0).stop 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 10 monclient: shutdown 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 msgr2=0x7f8428070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f842402b820 tx=0x7f84240046c0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 shutdown_connections 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f840c06d810 0x7f840c06fcd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428143890 0x7f842806daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84280b3a10 0x7f8428070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 --2- 172.21.15.87:0/975306524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f84280b3630 0x7f8428070350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 >> 172.21.15.87:0/975306524 conn(0x7f8428073120 msgr2=0x7f84280b6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 shutdown_connections 2024-07-27T14:41:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 -- 172.21.15.87:0/975306524 wait complete. 2024-07-27T14:41:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.537+0000 7f8430694640 1 librados: shutdown 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 -- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd834143890 msgr2=0x7fd8340ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd834143890 0x7fd8340ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 -- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8340b3630 msgr2=0x7fd8341cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8340b3630 0x7fd8341cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 10 monclient: _finish_hunting 0 2024-07-27T14:41:27.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 monclient: found mon.noname-b 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.539+0000 7fd839095640 1 -- 172.21.15.87:0/2338493558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8340afd50 con 0x7fd8340b3a10 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.541+0000 7fd839095640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c530 tx=0x7fd82400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.541+0000 7fd82a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8340b3a10 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.541+0000 7fd82a7fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824017070 con 0x7fd8340b3a10 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: dump: 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.552 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:27.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd824005ce0 con 0x7fd8340b3a10 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824007270 con 0x7fd8340b3a10 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82a7fc640 10 monclient: dump: 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:27.554 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd82bfff640 4 set_mon_vals no callback set 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 5 monclient: authenticate success, global_id 4372 2024-07-27T14:41:27.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 10 monclient: _renew_subs 2024-07-27T14:41:27.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8340affc0 con 0x7fd8340b3a10 2024-07-27T14:41:27.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:27.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 10 monclient: _renew_subs 2024-07-27T14:41:27.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.545+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8340ae830 con 0x7fd8340b3a10 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.546+0000 7fd83bafa640 1 librados: init done 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.547+0000 7fd83bafa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.547+0000 7fd83bafa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.547+0000 7fd83bafa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.547+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd834069180 con 0x7fd8340b3a10 2024-07-27T14:41:27.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.4 2024-07-27T14:41:27.559 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.564+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd82401a070 con 0x7fd8340b3a10 2024-07-27T14:41:27.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.565+0000 7fd82a7fc640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 0x7fd81806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:27.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.565+0000 7fd839896640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 0x7fd81806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:27.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.565+0000 7fd839896640 10 monclient: get_auth_request con 0x7fd81806d810 auth_method 0 2024-07-27T14:41:27.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.566+0000 7fd839896640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 0x7fd81806fcd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd830036000 tx=0x7fd830030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:27.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.566+0000 7fd82a7fc640 10 client.4372.objecter ms_handle_connect 0x7fd81806d810 2024-07-27T14:41:27.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.572+0000 7f606d718640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:27.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.572+0000 7f606d718640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:27.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.572+0000 7f606d718640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.572+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6068075ac0 con 0x7f60680b1c80 2024-07-27T14:41:27.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.715+0000 7f67a67cd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:27.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.715+0000 7f67a67cd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:27.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.715+0000 7f67a67cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T14:41:27.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.715+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f67a00b53b0 con 0x7f67a00b0010 2024-07-27T14:41:27.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.718+0000 7f679cff9640 1 -- 172.21.15.87:0/3634424312 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6794058fc0 con 0x7f67a00b0010 2024-07-27T14:41:27.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.718+0000 7f679cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-27T14:41:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.718+0000 7f679cff9640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout:77309411338 2024-07-27T14:41:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 10 librados: watch_flush enter 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 10 librados: watch_flush exit 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 20 client.4331.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 10 client.4331.objecter successfully canceled tick 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 msgr2=0x7f678c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 0x7f678c06fc80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67900015d0 tx=0x7f6790009040 comp rx=0 tx=0).stop 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 10 monclient: shutdown 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 msgr2=0x7f67a0072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f679400b4a0 tx=0x7f679400b960 comp rx=0 tx=0).stop 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 shutdown_connections 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f678c06d7c0 0x7f678c06fc80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67a0143140 0x7f67a01e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.720+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67a00b0010 0x7f67a0072160 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.721+0000 7f67a67cd640 1 --2- 172.21.15.87:0/3634424312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67a00678a0 0x7f67a01ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.721+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 >> 172.21.15.87:0/3634424312 conn(0x7f67a006b890 msgr2=0x7f67a013cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.721+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 shutdown_connections 2024-07-27T14:41:27.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.721+0000 7f67a67cd640 1 -- 172.21.15.87:0/3634424312 wait complete. 2024-07-27T14:41:27.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.721+0000 7f67a67cd640 1 librados: shutdown 2024-07-27T14:41:27.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411337 got 77309411338 for osd.6 2024-07-27T14:41:27.732 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.796+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7fd824056d10 con 0x7fd8340b3a10 2024-07-27T14:41:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.796+0000 7fd82a7fc640 10 client.4372.objecter ms_dispatch 0x7fd834000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.796+0000 7fd82a7fc640 3 client.4372.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.796+0000 7fd82a7fc640 3 client.4372.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:27.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.796+0000 7fd82a7fc640 20 client.4372.objecter dump_active .. 0 homeless 2024-07-27T14:41:27.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.810+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fd8340affc0 con 0x7fd8340b3a10 2024-07-27T14:41:27.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.810+0000 7fd82a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:27.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.810+0000 7fd82a7fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:27.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.812+0000 7f604f7fe640 1 -- 172.21.15.87:0/305170311 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6060059a40 con 0x7f60680b1c80 2024-07-27T14:41:27.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.812+0000 7f604f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-27T14:41:27.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.812+0000 7f604f7fe640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.813 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 10 librados: watch_flush enter 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 10 librados: watch_flush exit 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 20 client.4369.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 10 client.4369.objecter successfully canceled tick 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 msgr2=0x7f605406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 0x7f605406fcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6058036000 tx=0x7f6058030000 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 10 monclient: shutdown 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 msgr2=0x7f606813c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f606000b9c0 tx=0x7f606000be80 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 shutdown_connections 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f605406d810 0x7f605406fcd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60680b1c80 0x7f606813c740 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60680af970 0x7f606814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 --2- 172.21.15.87:0/305170311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6068072b70 0x7f606813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 >> 172.21.15.87:0/305170311 conn(0x7f6068068150 msgr2=0x7f60680b5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 shutdown_connections 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 -- 172.21.15.87:0/305170311 wait complete. 2024-07-27T14:41:27.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.814+0000 7f606d718640 1 librados: shutdown 2024-07-27T14:41:27.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.3 2024-07-27T14:41:27.826 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:27.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.992+0000 7fd83bafa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:27.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.992+0000 7fd83bafa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:27.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.992+0000 7fd83bafa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:27.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.992+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd8340aea00 con 0x7fd8340b3a10 2024-07-27T14:41:27.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.996+0000 7fd82a7fc640 1 -- 172.21.15.87:0/2338493558 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd824058c70 con 0x7fd8340b3a10 2024-07-27T14:41:27.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.996+0000 7fd82a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-27T14:41:27.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.996+0000 7fd82a7fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:27.997 INFO:teuthology.orchestra.run.smithi087.stdout:64424509449 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 10 librados: watch_flush enter 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 10 librados: watch_flush exit 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 20 client.4372.objecter shutdown clearing up homeless session... 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 10 client.4372.objecter successfully canceled tick 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 msgr2=0x7fd81806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:27.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 0x7fd81806fcd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd830036000 tx=0x7fd830030000 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 10 monclient: shutdown 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 msgr2=0x7fd8341db830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c530 tx=0x7fd82400c9f0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 shutdown_connections 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fd81806d810 0x7fd81806fcd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd834143890 0x7fd8340ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8340b3a10 0x7fd8341db830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 --2- 172.21.15.87:0/2338493558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8340b3630 0x7fd8341cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 >> 172.21.15.87:0/2338493558 conn(0x7fd834073120 msgr2=0x7fd834067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 shutdown_connections 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 -- 172.21.15.87:0/2338493558 wait complete. 2024-07-27T14:41:28.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:27.998+0000 7fd83bafa640 1 librados: shutdown 2024-07-27T14:41:28.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.0 2024-07-27T14:41:28.011 DEBUG:teuthology.parallel:result is None 2024-07-27T14:41:28.011 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-27T14:41:28.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.110+0000 7f2dba829640 10 monclient: get_monmap_and_config 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.110+0000 7f2dba829640 10 monclient: build_initial_monmap 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.110+0000 7f2dba829640 10 monclient: monmap: 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:28.111907+0000 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:28.111907+0000 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:28.111 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:28.112 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.111+0000 7f2dba829640 1 Processor -- start 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.111+0000 7f2dba829640 1 -- start start 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.111+0000 7f2dba829640 10 monclient: init 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-07-27T14:41:28.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2db40b0130 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2db40b49b0 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 0x7f2db40b2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-b con 0x7f2db41406b0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db40b5d00 con 0x7f2db40b0130 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db40b15f0 con 0x7f2db40b49b0 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2db40b0510 con 0x7f2db41406b0 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2dba829640 10 monclient(hunting): authenticate will time out at 1228.138388s 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2db3fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f2db40b0130 0x7f2db413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45338/0 (socket says 172.21.15.87:45338) 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.112+0000 7f2db3fff640 1 -- 172.21.15.87:0/3075612570 learned_addr learned my addr 172.21.15.87:0/3075612570 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.113+0000 7f2db8dc6640 1 --2- 172.21.15.87:0/3075612570 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 0x7f2db40b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.113+0000 7f2db8dc6640 10 monclient(hunting): get_auth_request con 0x7f2db41406b0 auth_method 0 2024-07-27T14:41:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.113+0000 7f2db8dc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.113+0000 7f2db8dc6640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.113+0000 7f2db8dc6640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.114+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.114+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.114+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929885662 0 0) 0x7f2db40b15f0 con 0x7f2db40b49b0 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _negotiate 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): my global_id is 4375 2024-07-27T14:41:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d9c003520 con 0x7f2db40b49b0 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877306632 0 0) 0x7f2db40b5d00 con 0x7f2db40b0130 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _negotiate 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 10 monclient(hunting): my global_id is 4356 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db40b15f0 con 0x7f2db40b0130 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 1 -- 172.21.15.87:0/3075612570 >> v1:172.21.15.112:6789/0 conn(0x7f2db40b49b0 legacy=0x7f2db40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 1 -- 172.21.15.87:0/3075612570 >> v1:172.21.15.87:6789/0 conn(0x7f2db40b0130 legacy=0x7f2db413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 10 monclient: _finish_hunting 0 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 1 monclient: found mon.noname-b 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:28.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.115+0000 7f2db8dc6640 1 -- 172.21.15.87:0/3075612570 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db40b61e0 con 0x7f2db41406b0 2024-07-27T14:41:28.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.118+0000 7f2db8dc6640 1 --2- 172.21.15.87:0/3075612570 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 0x7f2db40b2400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009dc0 tx=0x7f2dac02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc225e173818ab96 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac039070 con 0x7f2db41406b0 2024-07-27T14:41:28.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: dump: 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.122 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: _finish_auth 0 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: _check_auth_tickets 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dac02ccd0 con 0x7f2db41406b0 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2dba829640 5 monclient: authenticate success, global_id 4337 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2dba829640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2dba829640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 1 -- 172.21.15.87:0/3075612570 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac034660 con 0x7f2db41406b0 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2dba829640 4 set_mon_vals no callback set 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.121+0000 7f2db2ffd640 10 monclient: dump: 2024-07-27T14:41:28.123 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.124 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 10 monclient: shutdown 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 -- 172.21.15.87:0/3075612570 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 msgr2=0x7f2db40b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 --2- 172.21.15.87:0/3075612570 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 0x7f2db40b2400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009dc0 tx=0x7f2dac02bf20 comp rx=0 tx=0).stop 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 -- 172.21.15.87:0/3075612570 shutdown_connections 2024-07-27T14:41:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 --2- 172.21.15.87:0/3075612570 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db41406b0 0x7f2db40b2400 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 -- 172.21.15.87:0/3075612570 >> 172.21.15.87:0/3075612570 conn(0x7f2db40b8ff0 msgr2=0x7f2db40b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 -- 172.21.15.87:0/3075612570 shutdown_connections 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.122+0000 7f2dba829640 1 -- 172.21.15.87:0/3075612570 wait complete. 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 10 monclient: build_initial_monmap 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 10 monclient: monmap: 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:28.131 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 librados: starting msgr at 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 librados: starting objecter 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 Processor -- start 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 -- start start 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 librados: setting wanted keys 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 1 librados: calling monclient init 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.123+0000 7f2dba829640 10 monclient: init 2024-07-27T14:41:28.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db40b0130 0x7f2db4146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-c con 0x7f2db40b0130 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-a con 0x7f2db40b49b0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 0x7f2db4134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): picked mon.noname-b con 0x7f2db41406b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2db414e7c0 con 0x7f2db40b49b0 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2db414e640 con 0x7f2db41406b0 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2db414e940 con 0x7f2db40b0130 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2dba829640 10 monclient(hunting): authenticate will time out at 1228.150339s 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2db3fff640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db40b0130 0x7f2db4146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.124+0000 7f2db8dc6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 0x7f2db4134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43012/0 (socket says 172.21.15.87:43012) 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 1 -- 172.21.15.87:0/694645847 learned_addr learned my addr 172.21.15.87:0/694645847 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): get_auth_request con 0x7f2db40b49b0 auth_method 0 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): get_auth_request con 0x7f2db40b0130 auth_method 0 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): get_auth_request con 0x7f2db41406b0 auth_method 0 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.125+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 -- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db40b0130 msgr2=0x7f2db4146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db40b0130 0x7f2db4146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 -- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 msgr2=0x7f2db4133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 10 monclient: _finish_hunting 0 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 monclient: found mon.noname-b 2024-07-27T14:41:28.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:28.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:41:28.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db8dc6640 1 -- 172.21.15.87:0/694645847 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db4143c40 con 0x7f2db41406b0 2024-07-27T14:41:28.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.128+0000 7f2db37fe640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.130+0000 7f2db8dc6640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 0x7f2db4134290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dac033000 tx=0x7f2dac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.130+0000 7f2db0ff9640 10 client.?.objecter ms_handle_connect 0x7f2db41406b0 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.130+0000 7f2db0ff9640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac044020 con 0x7f2db41406b0 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: dump: 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: _finish_auth 0 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: _check_auth_tickets 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 5 monclient: authenticate success, global_id 4378 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dac03f470 con 0x7f2db41406b0 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 10 monclient: _renew_subs 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db41347d0 con 0x7f2db41406b0 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac039040 con 0x7f2db41406b0 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db0ff9640 10 monclient: dump: 2024-07-27T14:41:28.144 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 10 monclient: _renew_subs 2024-07-27T14:41:28.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2db27fc640 4 set_mon_vals no callback set 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db4146190 con 0x7f2db41406b0 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.134+0000 7f2dba829640 1 librados: init done 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.135+0000 7f2dba829640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.135+0000 7f2dba829640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.135+0000 7f2dba829640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:41:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.135+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db406ca20 con 0x7f2db41406b0 2024-07-27T14:41:28.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.379+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2dac047020 con 0x7f2db41406b0 2024-07-27T14:41:28.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.379+0000 7f2db0ff9640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 0x7f2da006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.379+0000 7f2db3fff640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 0x7f2da006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.379+0000 7f2db3fff640 10 monclient: get_auth_request con 0x7f2da006d7f0 auth_method 0 2024-07-27T14:41:28.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.380+0000 7f2db3fff640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 0x7f2da006fcb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2da4001200 tx=0x7f2da4000bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.380+0000 7f2db0ff9640 10 client.4378.objecter ms_handle_connect 0x7f2da006d7f0 2024-07-27T14:41:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.382+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f2dac079f90 con 0x7f2db41406b0 2024-07-27T14:41:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.382+0000 7f2db0ff9640 10 client.4378.objecter ms_dispatch 0x7f2db4000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.382+0000 7f2db0ff9640 3 client.4378.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.382+0000 7f2db0ff9640 3 client.4378.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.382+0000 7f2db0ff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-07-27T14:41:28.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.393+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f2db406ca20 con 0x7f2db41406b0 2024-07-27T14:41:28.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.393+0000 7f2db0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:28.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.393+0000 7f2db0ff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:28.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.612+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2db413ffa0 con 0x7f2da006d7f0 2024-07-27T14:41:28.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.613+0000 7f2db0ff9640 1 -- 172.21.15.87:0/694645847 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35777 (secure 0 0 0) 0x7f2db413ffa0 con 0x7f2da006d7f0 2024-07-27T14:41:28.614 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:28.614 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 10 librados: watch_flush enter 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 10 librados: watch_flush exit 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 20 client.4378.objecter shutdown clearing up homeless session... 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 10 client.4378.objecter successfully canceled tick 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 msgr2=0x7f2da006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 0x7f2da006fcb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2da4001200 tx=0x7f2da4000bc0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 10 monclient: shutdown 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 msgr2=0x7f2db4134290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 0x7f2db4134290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dac033000 tx=0x7f2dac004a40 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 shutdown_connections 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2da006d7f0 0x7f2da006fcb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db41406b0 0x7f2db4134290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2db40b49b0 0x7f2db4133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 --2- 172.21.15.87:0/694645847 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2db40b0130 0x7f2db4146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 >> 172.21.15.87:0/694645847 conn(0x7f2db40b8ff0 msgr2=0x7f2db413e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 shutdown_connections 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.615+0000 7f2dba829640 1 -- 172.21.15.87:0/694645847 wait complete. 2024-07-27T14:41:28.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.616+0000 7f2dba829640 1 librados: shutdown 2024-07-27T14:41:28.626 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-07-27T14:41:27.742297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218884,"kb_used_data":2712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755836,"statfs":{"total":773094113280,"available":772869976064,"internally_reserved":0,"allocated":2777088,"data_stored":1481246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.161616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.964981+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T14:41:21.964981+0000","last_peered":"2024-07-27T14:41:21.964981+0000","last_clean":"2024-07-27T14:41:21.964981+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T14:41:21.964981+0000","last_undegraded":"2024-07-27T14:41:21.964981+0000","last_fullsized":"2024-07-27T14:41:21.964981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:05:24.194010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.933661+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T14:41:21.933661+0000","last_peered":"2024-07-27T14:41:21.933661+0000","last_clean":"2024-07-27T14:41:21.933661+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T14:41:21.933661+0000","last_undegraded":"2024-07-27T14:41:21.933661+0000","last_fullsized":"2024-07-27T14:41:21.933661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.966051+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T14:41:21.966051+0000","last_peered":"2024-07-27T14:41:21.966051+0000","last_clean":"2024-07-27T14:41:21.966051+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T14:41:21.966051+0000","last_undegraded":"2024-07-27T14:41:21.966051+0000","last_fullsized":"2024-07-27T14:41:21.966051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:57:17.520382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.793384+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T14:41:21.793384+0000","last_peered":"2024-07-27T14:41:21.793384+0000","last_clean":"2024-07-27T14:41:21.793384+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T14:41:21.793384+0000","last_undegraded":"2024-07-27T14:41:21.793384+0000","last_fullsized":"2024-07-27T14:41:21.793384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.962086+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T14:41:21.962086+0000","last_peered":"2024-07-27T14:41:21.962086+0000","last_clean":"2024-07-27T14:41:21.962086+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T14:41:21.962086+0000","last_undegraded":"2024-07-27T14:41:21.962086+0000","last_fullsized":"2024-07-27T14:41:21.962086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:39:58.978046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.929977+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T14:41:21.929977+0000","last_peered":"2024-07-27T14:41:21.929977+0000","last_clean":"2024-07-27T14:41:21.929977+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T14:41:21.929977+0000","last_undegraded":"2024-07-27T14:41:21.929977+0000","last_fullsized":"2024-07-27T14:41:21.929977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.891242+0000","last_change":"2024-07-27T14:41:21.948784+0000","last_active":"2024-07-27T14:41:21.891242+0000","last_peered":"2024-07-27T14:41:21.891242+0000","last_clean":"2024-07-27T14:41:21.891242+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T14:41:21.891242+0000","last_undegraded":"2024-07-27T14:41:21.891242+0000","last_fullsized":"2024-07-27T14:41:21.891242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:23:48.411987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.976789+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T14:41:21.976789+0000","last_peered":"2024-07-27T14:41:21.976789+0000","last_clean":"2024-07-27T14:41:21.976789+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T14:41:21.976789+0000","last_undegraded":"2024-07-27T14:41:21.976789+0000","last_fullsized":"2024-07-27T14:41:21.976789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:40:29.539136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'76","reported_seq":129,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.900592+0000","last_change":"2024-07-27T14:41:03.617220+0000","last_active":"2024-07-27T14:41:21.900592+0000","last_peered":"2024-07-27T14:41:21.900592+0000","last_clean":"2024-07-27T14:41:21.900592+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T14:41:21.900592+0000","last_undegraded":"2024-07-27T14:41:21.900592+0000","last_fullsized":"2024-07-27T14:41:21.900592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_clean_scrub_stamp":"2024-07-27T14:41:00.649818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:14:15.632432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":85710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":85710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":483531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":483531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T14:41:28.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.725+0000 7f19d78ea640 10 monclient: get_monmap_and_config 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.725+0000 7f19d78ea640 10 monclient: build_initial_monmap 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.725+0000 7f19d78ea640 10 monclient: monmap: 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:28.726824+0000 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:28.726824+0000 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:28.727 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.726+0000 7f19d78ea640 1 Processor -- start 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.726+0000 7f19d78ea640 1 -- start start 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.726+0000 7f19d78ea640 10 monclient: init 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19d006bbb0 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f19d0146d50 0x7f19d006bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-c con 0x7f19d0146d50 addr v2:172.21.15.112:3300/0 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19d013fd90 addr v1:172.21.15.112:6789/0 2024-07-27T14:41:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d006d840 con 0x7f19d006bbb0 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d006c7b0 con 0x7f19d013fd90 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f19d01461b0 con 0x7f19d0146d50 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d78ea640 10 monclient(hunting): authenticate will time out at 1228.753473s 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d5686640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f19d006bbb0 0x7f19d0146640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45342/0 (socket says 172.21.15.87:45342) 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.727+0000 7f19d5686640 1 -- 172.21.15.87:0/1942334568 learned_addr learned my addr 172.21.15.87:0/1942334568 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.728+0000 7f19d4e85640 1 --2- 172.21.15.87:0/1942334568 >> v2:172.21.15.112:3300/0 conn(0x7f19d0146d50 0x7f19d006bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.728+0000 7f19d4e85640 10 monclient(hunting): get_auth_request con 0x7f19d0146d50 auth_method 0 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.728+0000 7f19d4e85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.728+0000 7f19d4e85640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.728+0000 7f19d4e85640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.729+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.729+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.729+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 1 -- 172.21.15.87:0/1942334568 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904385910 0 0) 0x7f19d006d840 con 0x7f19d006bbb0 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 10 monclient(hunting): _negotiate 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 10 monclient(hunting): my global_id is 4362 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.730+0000 7f19c7fff640 1 -- 172.21.15.87:0/1942334568 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19bc003520 con 0x7f19d006bbb0 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 1 -- 172.21.15.87:0/1942334568 >> v1:172.21.15.112:6789/0 conn(0x7f19d013fd90 legacy=0x7f19d0140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 1 -- 172.21.15.87:0/1942334568 >> v1:172.21.15.87:6789/0 conn(0x7f19d006bbb0 legacy=0x7f19d0146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 10 monclient: _finish_hunting 0 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 1 monclient: found mon.noname-c 2024-07-27T14:41:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:28.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T14:41:28.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.731+0000 7f19d4e85640 1 -- 172.21.15.87:0/1942334568 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d006cc00 con 0x7f19d0146d50 2024-07-27T14:41:28.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.735+0000 7f19d4e85640 1 --2- 172.21.15.87:0/1942334568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d0146d50 0x7f19d006bf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19c00097a0 tx=0x7f19c002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87fc7869009ae062 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 1 -- 172.21.15.87:0/1942334568 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0034070 con 0x7f19d0146d50 2024-07-27T14:41:28.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: dump: 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: _finish_auth 0 2024-07-27T14:41:28.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: _check_auth_tickets 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 1 -- 172.21.15.87:0/1942334568 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c0030610 con 0x7f19d0146d50 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19d78ea640 5 monclient: authenticate success, global_id 4381 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19d78ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 1 -- 172.21.15.87:0/1942334568 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c00309e0 con 0x7f19d0146d50 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19d78ea640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19d78ea640 4 set_mon_vals no callback set 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.738+0000 7f19c7fff640 10 monclient: dump: 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.741 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 10 monclient: shutdown 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 -- 172.21.15.87:0/1942334568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d0146d50 msgr2=0x7f19d006bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 --2- 172.21.15.87:0/1942334568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d0146d50 0x7f19d006bf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19c00097a0 tx=0x7f19c002b960 comp rx=0 tx=0).stop 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 -- 172.21.15.87:0/1942334568 shutdown_connections 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 --2- 172.21.15.87:0/1942334568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d0146d50 0x7f19d006bf90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 -- 172.21.15.87:0/1942334568 >> 172.21.15.87:0/1942334568 conn(0x7f19d0067ee0 msgr2=0x7f19d00b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:28.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 -- 172.21.15.87:0/1942334568 shutdown_connections 2024-07-27T14:41:28.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.739+0000 7f19d78ea640 1 -- 172.21.15.87:0/1942334568 wait complete. 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 10 monclient: build_initial_monmap 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 10 monclient: monmap: 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:28.746 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 librados: starting msgr at 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 librados: starting objecter 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 Processor -- start 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 -- start start 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 librados: setting wanted keys 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 1 librados: calling monclient init 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.740+0000 7f19d78ea640 10 monclient: init 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:28.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 0x7f19d0138300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-b con 0x7f19d006bbb0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19d013fd90 0x7f19d01db740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-c con 0x7f19d013fd90 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): picked mon.noname-a con 0x7f19d0146d50 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f19d014e620 con 0x7f19d0146d50 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f19d014e4a0 con 0x7f19d006bbb0 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f19d014e7a0 con 0x7f19d013fd90 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:28.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d78ea640 10 monclient(hunting): authenticate will time out at 1228.767344s 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d5e87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d5686640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 0x7f19d0138300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.741+0000 7f19d4e85640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19d013fd90 0x7f19d01db740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43024/0 (socket says 172.21.15.87:43024) 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 1 -- 172.21.15.87:0/3715384756 learned_addr learned my addr 172.21.15.87:0/3715384756 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): get_auth_request con 0x7f19d0146d50 auth_method 0 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): get_auth_request con 0x7f19d006bbb0 auth_method 0 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): get_auth_request con 0x7f19d013fd90 auth_method 0 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5e87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d5686640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.742+0000 7f19d4e85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-07-27T14:41:28.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 -- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19d013fd90 msgr2=0x7f19d01db740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19d013fd90 0x7f19d01db740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 -- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 msgr2=0x7f19d0138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 0x7f19d0138300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 10 monclient: _finish_hunting 0 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 monclient: found mon.noname-a 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5e87640 1 -- 172.21.15.87:0/3715384756 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d0070750 con 0x7f19d0146d50 2024-07-27T14:41:28.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.745+0000 7f19d5686640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 0x7f19d0138300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.747+0000 7f19d5e87640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19c800c710 tx=0x7f19c800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.747+0000 7f19c5ffb640 10 client.?.objecter ms_handle_connect 0x7f19d0146d50 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.747+0000 7f19c5ffb640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8016020 con 0x7f19d0146d50 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: dump: 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.753 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: _finish_auth 0 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: _check_auth_tickets 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c8006d50 con 0x7f19d0146d50 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 5 monclient: authenticate success, global_id 4365 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 10 monclient: _renew_subs 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:28.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d0144700 con 0x7f19d0146d50 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8003bb0 con 0x7f19d0146d50 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c5ffb640 10 monclient: dump: 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19c77fe640 4 set_mon_vals no callback set 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 10 monclient: _renew_subs 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d013f670 con 0x7f19d0146d50 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.751+0000 7f19d78ea640 1 librados: init done 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.752+0000 7f19d78ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.752+0000 7f19d78ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.752+0000 7f19d78ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:28.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.752+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d006bf90 con 0x7f19d0146d50 2024-07-27T14:41:28.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.771+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f19c800a850 con 0x7f19d0146d50 2024-07-27T14:41:28.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.771+0000 7f19c5ffb640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 0x7f19b406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:28.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.772+0000 7f19d5686640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 0x7f19b406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:28.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.772+0000 7f19d5686640 10 monclient: get_auth_request con 0x7f19b406d810 auth_method 0 2024-07-27T14:41:28.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.772+0000 7f19c5ffb640 10 client.4365.objecter ms_handle_connect 0x7f19b406d810 2024-07-27T14:41:28.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.772+0000 7f19d5686640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 0x7f19b406fcd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19cc005880 tx=0x7f19cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:28.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.774+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f19c8057c80 con 0x7f19d0146d50 2024-07-27T14:41:28.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.774+0000 7f19c5ffb640 10 client.4365.objecter ms_dispatch 0x7f19d0000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:28.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.774+0000 7f19c5ffb640 3 client.4365.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:28.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.774+0000 7f19c5ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:28.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:28.774+0000 7f19c5ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-07-27T14:41:29.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.016+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f19c8020020 con 0x7f19d0146d50 2024-07-27T14:41:29.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.016+0000 7f19c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:29.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.016+0000 7f19c5ffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:29.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.188+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f19d0075ac0 con 0x7f19b406d810 2024-07-27T14:41:29.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.189+0000 7f19c5ffb640 1 -- 172.21.15.87:0/3715384756 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35777 (secure 0 0 0) 0x7f19d0075ac0 con 0x7f19b406d810 2024-07-27T14:41:29.190 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:29.191 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T14:41:29.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 10 librados: watch_flush enter 2024-07-27T14:41:29.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 10 librados: watch_flush exit 2024-07-27T14:41:29.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 20 client.4365.objecter shutdown clearing up homeless session... 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 10 client.4365.objecter successfully canceled tick 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 msgr2=0x7f19b406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 0x7f19b406fcd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19cc005880 tx=0x7f19cc009040 comp rx=0 tx=0).stop 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 10 monclient: shutdown 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 msgr2=0x7f19d01dbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19c800c710 tx=0x7f19c800cbd0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 shutdown_connections 2024-07-27T14:41:29.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f19b406d810 0x7f19b406fcd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d0146d50 0x7f19d01dbc80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19d013fd90 0x7f19d01db740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 --2- 172.21.15.87:0/3715384756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19d006bbb0 0x7f19d0138300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.191+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 >> 172.21.15.87:0/3715384756 conn(0x7f19d0067ee0 msgr2=0x7f19d006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.192+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 shutdown_connections 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.192+0000 7f19d78ea640 1 -- 172.21.15.87:0/3715384756 wait complete. 2024-07-27T14:41:29.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.192+0000 7f19d78ea640 1 librados: shutdown 2024-07-27T14:41:29.203 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-07-27T14:41:27.742297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218884,"kb_used_data":2712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755836,"statfs":{"total":773094113280,"available":772869976064,"internally_reserved":0,"allocated":2777088,"data_stored":1481246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.161616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.964981+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T14:41:21.964981+0000","last_peered":"2024-07-27T14:41:21.964981+0000","last_clean":"2024-07-27T14:41:21.964981+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T14:41:21.964981+0000","last_undegraded":"2024-07-27T14:41:21.964981+0000","last_fullsized":"2024-07-27T14:41:21.964981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:05:24.194010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.933661+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T14:41:21.933661+0000","last_peered":"2024-07-27T14:41:21.933661+0000","last_clean":"2024-07-27T14:41:21.933661+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T14:41:21.933661+0000","last_undegraded":"2024-07-27T14:41:21.933661+0000","last_fullsized":"2024-07-27T14:41:21.933661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.966051+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T14:41:21.966051+0000","last_peered":"2024-07-27T14:41:21.966051+0000","last_clean":"2024-07-27T14:41:21.966051+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T14:41:21.966051+0000","last_undegraded":"2024-07-27T14:41:21.966051+0000","last_fullsized":"2024-07-27T14:41:21.966051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:57:17.520382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.793384+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T14:41:21.793384+0000","last_peered":"2024-07-27T14:41:21.793384+0000","last_clean":"2024-07-27T14:41:21.793384+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T14:41:21.793384+0000","last_undegraded":"2024-07-27T14:41:21.793384+0000","last_fullsized":"2024-07-27T14:41:21.793384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.962086+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T14:41:21.962086+0000","last_peered":"2024-07-27T14:41:21.962086+0000","last_clean":"2024-07-27T14:41:21.962086+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T14:41:21.962086+0000","last_undegraded":"2024-07-27T14:41:21.962086+0000","last_fullsized":"2024-07-27T14:41:21.962086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:39:58.978046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.929977+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T14:41:21.929977+0000","last_peered":"2024-07-27T14:41:21.929977+0000","last_clean":"2024-07-27T14:41:21.929977+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T14:41:21.929977+0000","last_undegraded":"2024-07-27T14:41:21.929977+0000","last_fullsized":"2024-07-27T14:41:21.929977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.891242+0000","last_change":"2024-07-27T14:41:21.948784+0000","last_active":"2024-07-27T14:41:21.891242+0000","last_peered":"2024-07-27T14:41:21.891242+0000","last_clean":"2024-07-27T14:41:21.891242+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T14:41:21.891242+0000","last_undegraded":"2024-07-27T14:41:21.891242+0000","last_fullsized":"2024-07-27T14:41:21.891242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:23:48.411987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.976789+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T14:41:21.976789+0000","last_peered":"2024-07-27T14:41:21.976789+0000","last_clean":"2024-07-27T14:41:21.976789+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T14:41:21.976789+0000","last_undegraded":"2024-07-27T14:41:21.976789+0000","last_fullsized":"2024-07-27T14:41:21.976789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:40:29.539136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'76","reported_seq":129,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-07-27T14:41:21.900592+0000","last_change":"2024-07-27T14:41:03.617220+0000","last_active":"2024-07-27T14:41:21.900592+0000","last_peered":"2024-07-27T14:41:21.900592+0000","last_clean":"2024-07-27T14:41:21.900592+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T14:41:21.900592+0000","last_undegraded":"2024-07-27T14:41:21.900592+0000","last_fullsized":"2024-07-27T14:41:21.900592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_clean_scrub_stamp":"2024-07-27T14:41:00.649818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:14:15.632432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":85710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":85710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":483531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":483531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":85691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T14:41:29.204 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-27T14:41:29.205 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-27T14:41:29.205 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-07-27T14:41:29.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-07-27T14:41:29.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.305+0000 7fc2dbbfb640 10 monclient: get_monmap_and_config 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.305+0000 7fc2dbbfb640 10 monclient: build_initial_monmap 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.305+0000 7fc2dbbfb640 10 monclient: monmap: 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:41:29.306659+0000 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:41:29.306659+0000 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T14:41:29.306 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T14:41:29.307 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T14:41:29.307 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T14:41:29.307 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T14:41:29.307 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 1 Processor -- start 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 1 -- start start 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 10 monclient: init 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 0x7fc2d40aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.306+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d4144200 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2d40b0450 addr v1:172.21.15.87:6789/0 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fc2d40710c0 0x7fc2d414b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d40710c0 addr v2:172.21.15.87:3300/0 2024-07-27T14:41:29.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4072c20 con 0x7fc2d40b0450 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fc2d41445e0 con 0x7fc2d40710c0 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d4144800 con 0x7fc2d4144200 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2dbbfb640 10 monclient(hunting): authenticate will time out at 1229.333046s 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2d9997640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 0x7fc2d40aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2d9196640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc2d40b0450 0x7fc2d40709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45348/0 (socket says 172.21.15.87:45348) 2024-07-27T14:41:29.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.307+0000 7fc2d9196640 1 -- 172.21.15.87:0/1696438830 learned_addr learned my addr 172.21.15.87:0/1696438830 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2da198640 1 --2- 172.21.15.87:0/1696438830 >> v2:172.21.15.87:3300/0 conn(0x7fc2d40710c0 0x7fc2d414b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): get_auth_request con 0x7fc2d4144200 auth_method 0 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2da198640 10 monclient(hunting): get_auth_request con 0x7fc2d40710c0 auth_method 0 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2da198640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2da198640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2da198640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:29.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.308+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.309+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.309+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.309+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 1 -- 172.21.15.87:0/1696438830 >> v2:172.21.15.87:3300/0 conn(0x7fc2d40710c0 msgr2=0x7fc2d414b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:41:29.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 1 --2- 172.21.15.87:0/1696438830 >> v2:172.21.15.87:3300/0 conn(0x7fc2d40710c0 0x7fc2d414b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 1 -- 172.21.15.87:0/1696438830 >> v1:172.21.15.87:6789/0 conn(0x7fc2d40b0450 legacy=0x7fc2d40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 10 monclient: _finish_hunting 0 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 1 monclient: found mon.noname-b 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T14:41:29.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.310+0000 7fc2d9997640 1 -- 172.21.15.87:0/1696438830 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d40706b0 con 0x7fc2d4144200 2024-07-27T14:41:29.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.312+0000 7fc2d9997640 1 --2- 172.21.15.87:0/1696438830 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 0x7fc2d40aff10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00097a0 tx=0x7fc2d002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad60c4bca2cdc56 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:29.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 1 -- 172.21.15.87:0/1696438830 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0035020 con 0x7fc2d4144200 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: dump: 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.317 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:29.318 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: _finish_auth 0 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: _check_auth_tickets 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 1 -- 172.21.15.87:0/1696438830 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d0031830 con 0x7fc2d4144200 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2dbbfb640 5 monclient: authenticate success, global_id 4346 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2dbbfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2dbbfb640 10 monclient: get_monmap_and_config success 2024-07-27T14:41:29.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2dbbfb640 4 set_mon_vals no callback set 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 1 -- 172.21.15.87:0/1696438830 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0031c00 con 0x7fc2d4144200 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.316+0000 7fc2d8995640 10 monclient: dump: 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.320 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:29.321 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 10 monclient: shutdown 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/1696438830 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 msgr2=0x7fc2d40aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/1696438830 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 0x7fc2d40aff10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00097a0 tx=0x7fc2d002b8d0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/1696438830 shutdown_connections 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/1696438830 >> v2:172.21.15.87:3300/0 conn(0x7fc2d40710c0 0x7fc2d414b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/1696438830 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d4144200 0x7fc2d40aff10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/1696438830 >> 172.21.15.87:0/1696438830 conn(0x7fc2d40b69a0 msgr2=0x7fc2d40b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/1696438830 shutdown_connections 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.317+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/1696438830 wait complete. 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 10 monclient: build_initial_monmap 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 10 monclient: monmap: 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:41:29.327 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 librados: starting msgr at 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 librados: starting objecter 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 Processor -- start 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 -- start start 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 librados: setting wanted keys 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 1 librados: calling monclient init 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.318+0000 7fc2dbbfb640 10 monclient: init 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient: _reopen_session rank -1 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d40710c0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 0x7fc2d41398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d40b0450 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:41:29.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2d4144200 0x7fc2d4139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d4144200 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4144d20 con 0x7fc2d40710c0 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4073320 con 0x7fc2d4144200 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): start opening mon connection 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d41409e0 con 0x7fc2d40b0450 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): _renew_subs 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2dbbfb640 10 monclient(hunting): authenticate will time out at 1229.345193s 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9196640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 0x7fc2d41398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43038/0 (socket says 172.21.15.87:43038) 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 1 -- 172.21.15.87:0/413784169 learned_addr learned my addr 172.21.15.87:0/413784169 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 10 monclient(hunting): get_auth_request con 0x7fc2d40710c0 auth_method 0 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9997640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.319+0000 7fc2d9196640 10 monclient(hunting): get_auth_request con 0x7fc2d40b0450 auth_method 0 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:29.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2da198640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2d4144200 0x7fc2d4139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9997640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2d9196640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2da198640 10 monclient(hunting): get_auth_request con 0x7fc2d4144200 auth_method 0 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2da198640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2da198640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.320+0000 7fc2da198640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.321+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:41:29.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.321+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.321+0000 7fc2da198640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 -- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 msgr2=0x7fc2d41398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 0x7fc2d41398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 -- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2d4144200 msgr2=0x7fc2d4139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2d4144200 0x7fc2d4139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 10 monclient: _finish_hunting 0 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 monclient: found mon.noname-a 2024-07-27T14:41:29.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9997640 1 -- 172.21.15.87:0/413784169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d40ad270 con 0x7fc2d40710c0 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.323+0000 7fc2d9196640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 0x7fc2d41398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.325+0000 7fc2d9997640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00097a0 tx=0x7fc2d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.325+0000 7fc2ca7fc640 10 client.?.objecter ms_handle_connect 0x7fc2d40710c0 2024-07-27T14:41:29.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.325+0000 7fc2ca7fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d003a970 con 0x7fc2d40710c0 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: dump: 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:29.335 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: _finish_auth 0 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: _check_auth_tickets 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d003ab10 con 0x7fc2d40710c0 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 5 monclient: authenticate success, global_id 4374 2024-07-27T14:41:29.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 10 monclient: _renew_subs 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d41cdd30 con 0x7fc2d40710c0 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00443f0 con 0x7fc2d40710c0 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2ca7fc640 10 monclient: dump: 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T14:41:29.337 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2cbfff640 4 set_mon_vals no callback set 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 10 monclient: _renew_subs 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:29.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d40acf00 con 0x7fc2d40710c0 2024-07-27T14:41:29.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.329+0000 7fc2dbbfb640 1 librados: init done 2024-07-27T14:41:29.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.330+0000 7fc2dbbfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:29.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.330+0000 7fc2dbbfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:29.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.330+0000 7fc2dbbfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:29.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.330+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d40aff10 con 0x7fc2d40710c0 2024-07-27T14:41:29.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.348+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc2d004b060 con 0x7fc2d40710c0 2024-07-27T14:41:29.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.348+0000 7fc2ca7fc640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 0x7fc2b806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:41:29.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.349+0000 7fc2d9196640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 0x7fc2b806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:41:29.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.349+0000 7fc2d9196640 10 monclient: get_auth_request con 0x7fc2b806d810 auth_method 0 2024-07-27T14:41:29.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.349+0000 7fc2d9196640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 0x7fc2b806fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40ad4d0 tx=0x7fc2c4007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:41:29.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.349+0000 7fc2ca7fc640 10 client.4374.objecter ms_handle_connect 0x7fc2b806d810 2024-07-27T14:41:29.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.352+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7fc2d0078dd0 con 0x7fc2d40710c0 2024-07-27T14:41:29.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.352+0000 7fc2ca7fc640 10 client.4374.objecter ms_dispatch 0x7fc2d4000b90 osd_map(25..25 src has 1..25) 2024-07-27T14:41:29.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.352+0000 7fc2ca7fc640 3 client.4374.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:41:29.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.352+0000 7fc2ca7fc640 3 client.4374.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:41:29.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.352+0000 7fc2ca7fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-07-27T14:41:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.609+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fc2d003f070 con 0x7fc2d40710c0 2024-07-27T14:41:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.609+0000 7fc2ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T14:41:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.609+0000 7fc2ca7fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T14:41:29.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.856+0000 7fc2dbbfb640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-07-27T14:41:29.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.856+0000 7fc2dbbfb640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-07-27T14:41:29.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.856+0000 7fc2dbbfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T14:41:29.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.856+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc2d40af600 con 0x7fc2d40710c0 2024-07-27T14:41:29.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.862+0000 7fc2ca7fc640 1 -- 172.21.15.87:0/413784169 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc2d0031800 con 0x7fc2d40710c0 2024-07-27T14:41:29.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.862+0000 7fc2ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-07-27T14:41:29.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.862+0000 7fc2ca7fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T14:41:29.863 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T14:41:29.863 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-27T14:41:29.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 10 librados: watch_flush enter 2024-07-27T14:41:29.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 10 librados: watch_flush exit 2024-07-27T14:41:29.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 20 client.4374.objecter shutdown clearing up homeless session... 2024-07-27T14:41:29.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 10 client.4374.objecter successfully canceled tick 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 msgr2=0x7fc2b806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 0x7fc2b806fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40ad4d0 tx=0x7fc2c4007e80 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 10 monclient: shutdown 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 msgr2=0x7fc2d4141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00097a0 tx=0x7fc2d00046c0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 shutdown_connections 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc2b806d810 0x7fc2b806fcd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2d4144200 0x7fc2d4139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2d40b0450 0x7fc2d41398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 --2- 172.21.15.87:0/413784169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2d40710c0 0x7fc2d4141170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 >> 172.21.15.87:0/413784169 conn(0x7fc2d40b69a0 msgr2=0x7fc2d406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:41:29.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 shutdown_connections 2024-07-27T14:41:29.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 -- 172.21.15.87:0/413784169 wait complete. 2024-07-27T14:41:29.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T14:41:29.864+0000 7fc2dbbfb640 1 librados: shutdown 2024-07-27T14:41:29.876 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-07-27T14:41:29.877 INFO:teuthology.run_tasks:Running task workunit... 2024-07-27T14:41:29.894 INFO:tasks.workunit:Pulling workunits from ref 57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:41:29.894 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-27T14:41:29.895 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-27T14:41:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T14:41:29.911 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-07-27T14:41:29.911 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-07-27T14:41:29.966 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-07-27T14:41:29.966 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-07-27T14:41:30.021 INFO:tasks.workunit:timeout=6h 2024-07-27T14:41:30.021 INFO:tasks.workunit:cleanup=True 2024-07-27T14:41:30.022 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 57fd7103c78964675e1941b32d58041fb7c15af7 2024-07-27T14:41:30.077 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-27T14:41:31.328 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:34.195 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.331 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:34.198 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.372 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:34.238 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.380 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:34.232 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.381 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:34.234 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.389 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:34.256 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.390 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:34.257 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.419 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:34.271 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:31.436 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:34.303 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.324 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:54.190 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.372 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:54.238 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.373 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:54.225 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.376 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:54.229 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.390 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:01:54.243 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.392 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:54.259 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.422 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:54.288 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.425 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:54.291 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:41:51.439 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:54.306 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.456 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:14.322 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.461 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:14.328 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.481 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:14.348 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.511 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:14.363 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.512 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:14.364 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.526 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:14.378 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.543 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:14.409 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.545 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:14.412 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:11.595 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:14.462 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to '57fd7103c78964675e1941b32d58041fb7c15af7'. 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.686 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-07-27T14:42:25.687 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at 57fd7103c78 Merge pull request #58696 from zdover23/wip-doc-2024-07-21-backport-58695-to-squid 2024-07-27T14:42:25.690 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-27T14:42:25.768 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-27T14:42:25.770 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-27T14:42:25.770 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-27T14:42:26.334 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-27T14:42:26.387 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-27T14:42:26.426 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-27T14:42:26.427 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-27T14:42:26.427 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-27T14:42:26.466 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-27T14:42:26.469 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-07-27T14:42:26.469 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-27T14:42:26.525 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-07-27T14:42:26.527 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-07-27T14:42:26.527 DEBUG:teuthology.orchestra.run.smithi087:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57fd7103c78964675e1941b32d58041fb7c15af7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-07-27T14:42:26.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ parallel=1 2024-07-27T14:42:26.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = --serial ']' 2024-07-27T14:42:26.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ crimson=0 2024-07-27T14:42:26.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = --crimson ']' 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ color= 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -t 1 ']' 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ declare -A pids 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_aio 2024-07-27T14:42:26.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_aio' 2024-07-27T14:42:26.588 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_aio 2024-07-27T14:42:26.588 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.588 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_aio 2024-07-27T14:42:26.588 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40911 2024-07-27T14:42:26.589 INFO:tasks.workunit.client.0.smithi087.stdout:test api_aio on pid 40911 2024-07-27T14:42:26.591 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_aio on pid 40911' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40911 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_aio_pp 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_aio_pp' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_aio_pp 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.592 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.593 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_aio_pp 2024-07-27T14:42:26.594 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40918 2024-07-27T14:42:26.595 INFO:tasks.workunit.client.0.smithi087.stdout:test api_aio_pp on pid 40918 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_aio_pp on pid 40918' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40918 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_io 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_io' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_io 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.600 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.601 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.602 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_io 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40927 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_io on pid 40927' 2024-07-27T14:42:26.603 INFO:tasks.workunit.client.0.smithi087.stdout:test api_io on pid 40927 2024-07-27T14:42:26.606 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40927 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_io_pp 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_io_pp' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_io_pp 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.607 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.608 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.609 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.610 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_io_pp 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40938 2024-07-27T14:42:26.611 INFO:tasks.workunit.client.0.smithi087.stdout:test api_io_pp on pid 40938 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_io_pp on pid 40938' 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40938 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_asio 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_asio' 2024-07-27T14:42:26.615 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_asio 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.616 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.617 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.618 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.619 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.620 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_asio 2024-07-27T14:42:26.621 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40952 2024-07-27T14:42:26.621 INFO:tasks.workunit.client.0.smithi087.stdout:test api_asio on pid 40952 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_asio on pid 40952' 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40952 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_list 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_list' 2024-07-27T14:42:26.628 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_list 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.629 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.630 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.631 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.632 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.633 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.634 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.635 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_list 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40969 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_list on pid 40969' 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40911/exe 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.636 INFO:tasks.workunit.client.0.smithi087.stdout:test api_list on pid 40969 2024-07-27T14:42:26.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40969 2024-07-27T14:42:26.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_lock 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_lock' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_lock 2024-07-27T14:42:26.645 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.646 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.647 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.648 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.649 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.650 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.651 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_aio.log 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.652 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_aio: /' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40918/exe 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_lock 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40991 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_lock on pid 40991' 2024-07-27T14:42:26.653 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.654 INFO:tasks.workunit.client.0.smithi087.stdout:test api_lock on pid 40991 2024-07-27T14:42:26.665 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=40991 2024-07-27T14:42:26.665 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.665 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.665 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_lock_pp 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_lock_pp' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.666 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.667 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.668 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.669 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.670 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_aio_pp: /' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.671 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40927/exe 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.672 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.673 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_lock_pp 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.674 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.675 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.676 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_io: /' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_lock_pp 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.677 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.678 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.678 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.678 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.678 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41017 2024-07-27T14:42:26.678 INFO:tasks.workunit.client.0.smithi087.stdout:test api_lock_pp on pid 41017 2024-07-27T14:42:26.685 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_lock_pp on pid 41017' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41017 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.686 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.687 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40938/exe 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_misc 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_misc' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.688 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_misc 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_io.log 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.689 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.690 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.691 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.692 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_misc 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.693 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41036 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_misc on pid 41036' 2024-07-27T14:42:26.694 INFO:tasks.workunit.client.0.smithi087.stdout:test api_misc on pid 41036 2024-07-27T14:42:26.704 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41036 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_misc_pp 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_misc_pp' 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_misc_pp 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-07-27T14:42:26.705 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.706 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.707 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.709 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40952/exe 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.712 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_io_pp: /' 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.713 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.714 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.715 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_asio.log 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_misc_pp 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41062 2024-07-27T14:42:26.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_misc_pp on pid 41062' 2024-07-27T14:42:26.717 INFO:tasks.workunit.client.0.smithi087.stdout:test api_misc_pp on pid 41062 2024-07-27T14:42:26.731 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41062 2024-07-27T14:42:26.731 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.732 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.733 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_tier_pp 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.734 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_tier_pp' 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.735 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.736 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.737 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.738 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41017/exe 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.739 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40969/exe 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/40991/exe 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.741 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_lock_pp: /' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_asio: /' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_list: /' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_lock.log 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_tier_pp 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_tier_pp 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41091 2024-07-27T14:42:26.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_tier_pp on pid 41091' 2024-07-27T14:42:26.745 INFO:tasks.workunit.client.0.smithi087.stdout:test api_tier_pp on pid 41091 2024-07-27T14:42:26.752 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41091 2024-07-27T14:42:26.752 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_pool 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_pool' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.753 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_pool 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.755 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.756 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.757 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_lock: /' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_list.log 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.758 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_pool 2024-07-27T14:42:26.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41117 2024-07-27T14:42:26.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_pool on pid 41117' 2024-07-27T14:42:26.760 INFO:tasks.workunit.client.0.smithi087.stdout:test api_pool on pid 41117 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41117 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.768 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.769 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.770 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_snapshots 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_snapshots' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_snapshots 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.771 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_snapshots 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41147 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_snapshots on pid 41147' 2024-07-27T14:42:26.772 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.773 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.773 INFO:tasks.workunit.client.0.smithi087.stdout:test api_snapshots on pid 41147 2024-07-27T14:42:26.781 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41147 2024-07-27T14:42:26.781 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.781 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.782 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_snapshots_pp 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_snapshots_pp' 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.783 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.784 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41036/exe 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_snapshots_pp 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.787 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_snapshots_pp 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.788 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41168 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_snapshots_pp on pid 41168' 2024-07-27T14:42:26.789 INFO:tasks.workunit.client.0.smithi087.stdout:test api_snapshots_pp on pid 41168 2024-07-27T14:42:26.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41168 2024-07-27T14:42:26.798 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.798 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_stat 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_misc: /' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_stat' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_misc.log 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.800 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.801 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.802 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_stat 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.804 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.805 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.806 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_stat 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41191 2024-07-27T14:42:26.807 INFO:tasks.workunit.client.0.smithi087.stdout:test api_stat on pid 41191 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_stat on pid 41191' 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41191 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_stat_pp 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_stat_pp' 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41062/exe 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-07-27T14:42:26.827 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_misc_pp: /' 2024-07-27T14:42:26.828 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.830 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.832 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41091/exe 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.833 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.835 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_tier_pp: /' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.837 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.838 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.839 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41168/exe 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.840 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.841 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.842 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41117/exe 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-07-27T14:42:26.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_pool: /' 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_stat_pp 2024-07-27T14:42:26.844 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_stat_pp 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41276 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_stat_pp on pid 41276' 2024-07-27T14:42:26.846 INFO:tasks.workunit.client.0.smithi087.stdout:test api_stat_pp on pid 41276 2024-07-27T14:42:26.855 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41276 2024-07-27T14:42:26.855 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.855 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.855 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-07-27T14:42:26.855 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_watch_notify 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_watch_notify' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.856 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.857 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41147/exe 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.858 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.859 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.860 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_pool.log 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_snapshots: /' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_watch_notify 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_watch_notify 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41298 2024-07-27T14:42:26.862 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_watch_notify on pid 41298' 2024-07-27T14:42:26.863 INFO:tasks.workunit.client.0.smithi087.stdout:test api_watch_notify on pid 41298 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41298 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_watch_notify_pp 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-07-27T14:42:26.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_watch_notify_pp' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.867 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-07-27T14:42:26.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] Global test environment set-up. 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] 12 tests from AsioRados 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.732+0000 7f28f3e3a780 10 monclient: get_monmap_and_config 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.732+0000 7f28f3e3a780 10 monclient: build_initial_monmap 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.732+0000 7f28f3e3a780 10 monclient: monmap: 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 0 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 0.000000 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 0.000000 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 0 (unknown) 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.733+0000 7f28f3e3a780 1 Processor -- start 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.733+0000 7f28f3e3a780 1 -- start start 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.733+0000 7f28f3e3a780 10 monclient: init 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient: _reopen_session rank -1 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28dc002700 0x56045874e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-c con 0x7f28dc002700 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-a con 0x560458750d40 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604586c8320 0x56045874cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-b con 0x5604586c8320 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560458637dc0 con 0x560458750d40 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604585d74c0 con 0x5604586c8320 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x5604586bee30 con 0x7f28dc002700 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): _renew_subs 2024-07-27T14:42:26.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.734+0000 7f28f3e3a780 10 monclient(hunting): authenticate will time out at 1286.760224s 2024-07-27T14:42:26.875 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.875 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.875 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.875 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.875 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.876 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.877 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.878 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_watch_notify_pp 2024-07-27T14:42:26.879 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_watch_notify_pp 2024-07-27T14:42:26.879 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41332 2024-07-27T14:42:26.879 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_watch_notify_pp on pid 41332' 2024-07-27T14:42:26.879 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-2test api_watch_notify_pp on pid 41332 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41332 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.882 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.883 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41191/exe 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stdout:7T14:42:26.739+0000 7f28f0887640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52186/0 (socket says 172.21.15.87:52186) 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 1 -- 172.21.15.87:0/1496602056 learned_addr learned my addr 172.21.15.87:0/1496602056 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:42:26.884 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 10 monclient(hunting): get_auth_request con 0x560458750d40 auth_method 0 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.739+0000 7f28f0887640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.740+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.740+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.740+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 -- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28dc002700 msgr2=0x56045874e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28dc002700 0x56045874e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 -- 172.21.15.87:0/1496602056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604586c8320 msgr2=0x56045874cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604586c8320 0x56045874cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 10 monclient: _finish_hunting 0 2024-07-27T14:42:26.885 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 monclient: found mon.noname-a 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.742+0000 7f28f0887640 1 -- 172.21.15.87:0/1496602056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56045874fc40 con 0x560458750d40 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.745+0000 7f28f0887640 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e80449a0 tx=0x7f28e804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe2f2aa99c971db server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 1 -- 172.21.15.87:0/1496602056 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8061070 con 0x560458750d40 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: dump: 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 1 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.886 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 19 (squid) 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: _finish_auth 0 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: _check_auth_tickets 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 1 -- 172.21.15.87:0/1496602056 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e80527c0 con 0x560458750d40 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: handle_config config(0 keys) 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 1 -- 172.21.15.87:0/1496602056 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8057040 con 0x560458750d40 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:42:26.887 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28e37fe640 10 monclient: dump: 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 1 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 19 (squid) 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28f3e3a780 5 monclient: authenticate success, global_id 4401 2024-07-27T14:42:26.888 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28f3e3a780 10 monclient: get_monmap_and_config success 2024-07-27T14:42:26.889 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.762+0000 7f28f3e3a780 4 set_mon_vals no callback set 2024-07-27T14:42:26.889 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 10 monclient: shutdown 2024-07-27T14:42:26.889 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:42:26.889 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 -- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 msgr2=0x56045874f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:42:26.889 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e80449a0 tx=0x7f28e804c000 comp rx=0 tx=0).stop 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.891 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 -- 172.21.15.87:0/1496602056 shutdown_connections 2024-07-27T14:42:26.892 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604586c8320 0x56045874cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560458750d40 0x56045874f700 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/1496602056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28dc002700 0x56045874e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 -- 172.21.15.87:0/1496602056 >> 172.21.15.87:0/1496602056 conn(0x5604586c8e50 msgr2=0x56045874eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 -- 172.21.15.87:0/1496602056 shutdown_connections 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 1 -- 172.21.15.87:0/1496602056 wait complete. 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.763+0000 7f28f3e3a780 10 monclient: build_initial_monmap 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 10 monclient: monmap: 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 0 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 0.000000 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 0.000000 2024-07-27T14:42:26.893 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 0 (unknown) 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 librados: starting msgr at 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 librados: starting objecter 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 Processor -- start 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 -- start start 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 librados: setting wanted keys 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 1 librados: calling monclient init 2024-07-27T14:42:26.894 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.764+0000 7f28f3e3a780 10 monclient: init 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient: _reopen_session rank -1 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x5604586c8320 0x5604587f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-c con 0x5604586c8320 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 0x5604587f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.895 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-b con 0x560458750d40 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T14:42:26.897 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_cmd 2024-07-27T14:42:26.897 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_cmd' 2024-07-27T14:42:26.897 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.897 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_cmd 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_stat.log 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_stat: /' 2024-07-27T14:42:26.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_cmd 2024-07-27T14:42:26.899 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41370 2024-07-27T14:42:26.899 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_cmd on pid 41370' 2024-07-27T14:42:26.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28dc002700 0x56045874b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 ttest api_cmd on pid 41370 2024-07-27T14:42:26.901 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41370 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.902 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.903 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_cmd_pp 2024-07-27T14:42:26.904 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_cmd_pp' 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout:x=0 comp rx=0 tx=0).connect 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): picked mon.noname-a con 0x7f28dc002700 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5604586c9370 con 0x7f28dc002700 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604587dfb30 con 0x560458750d40 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): start opening mon connection 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x560458686780 con 0x5604586c8320 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): _renew_subs 2024-07-27T14:42:26.905 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.765+0000 7f28f3e3a780 10 monclient(hunting): authenticate will time out at 1286.791570s 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 0x5604587f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 0x5604587f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:59316/0 (socket says 172.21.15.87:59316) 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 1 -- 172.21.15.87:0/403720986 learned_addr learned my addr 172.21.15.87:0/403720986 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 10 monclient(hunting): get_auth_request con 0x560458750d40 auth_method 0 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f0887640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.776+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28dc002700 0x56045874b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.777+0000 7f28f1889640 10 monclient(hunting): get_auth_request con 0x7f28dc002700 auth_method 0 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.777+0000 7f28f1889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.777+0000 7f28f1889640 10 monclient(hunting): _init_auth method 2 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.777+0000 7f28f1889640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T14:42:26.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T14:42:26.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:42:26.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f0887640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.913 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.914 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_cmd_pp 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.915 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.916 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.917 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_cmd_pp 2024-07-27T14:42:26.918 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41420 2024-07-27T14:42:26.919 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.919 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_cmd_pp on pid 41420' 2024-07-27T14:42:26.919 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f1889640 10 monclient(hunting): handle_auth_repltest api_cmd_pp on pid 41420 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41420 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_service 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.928 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_service' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.929 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_service 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.930 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.931 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.932 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.933 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.934 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41332/exe 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.935 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.936 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_service 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41439 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_service on pid 41439' 2024-07-27T14:42:26.937 INFO:tasks.workunit.client.0.smithi087.stdout:test api_service on pid 41439 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41439 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41276/exe 2024-07-27T14:42:26.944 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_service_pp 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.945 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.946 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_service_pp' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.947 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_service_pp 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.948 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_service_pp 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_stat_pp: /' 2024-07-27T14:42:26.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41457 2024-07-27T14:42:26.950 INFO:tasks.workunit.client.0.smithi087.stdout:test api_service_pp on pid 41457 2024-07-27T14:42:26.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_service_pp on pid 41457' 2024-07-27T14:42:26.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41457 2024-07-27T14:42:26.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout:y_more payload 9 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f1889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.778+0000 7f28f1889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x5604586c8320 msgr2=0x5604587f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x5604586c8320 0x5604587f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28dc002700 msgr2=0x56045874b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28dc002700 0x56045874b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 10 monclient: _finish_hunting 0 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 monclient: found mon.noname-b 2024-07-27T14:42:26.951 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.782+0000 7f28f0887640 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560458686a60 con 0x560458750d40 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.788+0000 7f28f0887640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 0x5604587f60f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f28e8055040 tx=0x7f28dc003110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.788+0000 7f28e27fc640 10 client.?.objecter ms_handle_connect 0x560458750d40 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.788+0000 7f28e27fc640 10 client.?.objecter resend_mon_ops 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e806e070 con 0x560458750d40 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: dump: 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 1 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:26.952 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 19 (squid) 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.953 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: _finish_auth 0 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_c_write_operations 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_c_write_operations' 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-07-27T14:42:26.965 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.966 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.967 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.968 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_c_write_operations 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:26.969 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.970 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.971 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.972 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_c_write_operations 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.973 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41474 2024-07-27T14:42:26.974 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-2test api_c_write_operations on pid 41474 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_c_write_operations on pid 41474' 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41474 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.988 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.989 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41298/exe 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.990 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.991 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.992 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.993 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:26.994 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41420/exe 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_c_read_operations 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_c_read_operations' 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:26.995 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.996 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41370/exe 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:26.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_watch_notify: /' 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:26.998 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_cmd.log 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:26.999 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-07-27T14:42:27.000 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_cmd: /' 2024-07-27T14:42:27.001 INFO:tasks.workunit.client.0.smithi087.stdout:7T14:42:26.799+0000 7f28e27fc640 10 monclient: _check_auth_tickets 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8053e50 con 0x560458750d40 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: handle_config config(0 keys) 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8061040 con 0x560458750d40 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e27fc640 10 monclient: dump: 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: epoch 1 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: created 2024-07-27T14:39:24.184672+0000 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: min_mon_release 19 (squid) 2024-07-27T14:42:27.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: election_strategy: 1 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28e37fe640 4 set_mon_vals no callback set 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 5 monclient: authenticate success, global_id 4417 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 monclient: _renew_subs 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56045874c7f0 con 0x560458750d40 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 monclient: _renew_subs 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604586bfa10 con 0x560458750d40 2024-07-27T14:42:27.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 1 librados: init done 2024-07-27T14:42:27.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.799+0000 7f28f3e3a780 10 librados: wait_for_osdmap waiting 2024-07-27T14:42:27.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.826+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f28e8071070 con 0x560458750d40 2024-07-27T14:42:27.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.826+0000 7f28e27fc640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f28d406ba60 0x7f28d406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:27.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.827+0000 7f28f1088640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f28d406ba60 0x7f28d406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:42:27.005 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.006 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_c_read_operations 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_c_read_operations 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41520 2024-07-27T14:42:27.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.827+0000 7f28f10test api_c_read_operations on pid 41520 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_c_read_operations on pid 41520' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41520 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.016 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_cls_remote_reads 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_cls_remote_reads' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.020 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.022 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_cls_remote_reads 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41439/exe 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_cls_remote_reads 2024-07-27T14:42:27.023 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41565 2024-07-27T14:42:27.024 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_cls_remote_reads on pid 41565' 2024-07-27T14:42:27.024 INFO:tasks.workunit.client.0.smithi087.stdout:test api_cls_remote_reads on pid 41565 2024-07-27T14:42:27.039 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41565 2024-07-27T14:42:27.039 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:27.039 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.039 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.040 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.041 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.042 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41457/exe 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.043 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.044 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.045 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s list_parallel 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.046 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' list_parallel' 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.047 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_service.log 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_service: /' 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.048 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.049 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.050 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.051 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_service_pp: /' 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41474/exe 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo list_parallel 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.052 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.054 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=list_parallel 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41631 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test list_parallel on pid 41631' 2024-07-27T14:42:27.055 INFO:tasks.workunit.client.0.smithi087.stdout:test list_parallel on pid 41631 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41631 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-07-27T14:42:27.059 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s open_pools_parallel 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' open_pools_parallel' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo open_pools_parallel 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.060 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.061 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=open_pools_parallel 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41650 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test open_pools_parallel on pid 41650' 2024-07-27T14:42:27.063 INFO:tasks.workunit.client.0.smithi087.stdout:test open_pools_parallel on pid 41650 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41650 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.071 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s delete_pools_parallel 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' delete_pools_parallel' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo delete_pools_parallel 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.072 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.073 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.074 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.075 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.076 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.077 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=delete_pools_parallel 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.078 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41663 2024-07-27T14:42:27.079 INFO:tasks.workunit.client.0.smithi087.stdout:test delete_pools_parallel on pid 41663 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test delete_pools_parallel on pid 41663' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41663 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.088 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s cls 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' cls' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.089 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.090 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41520/exe 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41565/exe 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.091 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.092 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.093 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.094 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-07-27T14:42:27.095 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.096 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo cls 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=cls 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41696 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test cls on pid 41696' 2024-07-27T14:42:27.097 INFO:tasks.workunit.client.0.smithi087.stdout:test cls on pid 41696 2024-07-27T14:42:27.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41696 2024-07-27T14:42:27.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.100 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s cmd 2024-07-27T14:42:27.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' cmd' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo cmd 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.101 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.102 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=cmd 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41702 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test cmd on pid 41702' 2024-07-27T14:42:27.103 INFO:tasks.workunit.client.0.smithi087.stdout:test cmd on pid 41702 2024-07-27T14:42:27.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41702 2024-07-27T14:42:27.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-07-27T14:42:27.117 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.118 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s handler_error 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' handler_error' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.119 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.120 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.121 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.122 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.124 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.125 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.128 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo handler_error 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.129 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=handler_error 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41729 2024-07-27T14:42:27.130 INFO:tasks.workunit.client.0.smithi087.stdout:test handler_error on pid 41729 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test handler_error on pid 41729' 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41729 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s io 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' io' 2024-07-27T14:42:27.136 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41650/exe 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41631/exe 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.137 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.138 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo io 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=io 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41754 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test io on pid 41754' 2024-07-27T14:42:27.141 INFO:tasks.workunit.client.0.smithi087.stdout:test io on pid 41754 2024-07-27T14:42:27.147 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41754 2024-07-27T14:42:27.147 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.147 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.147 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.148 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s ec_io 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' ec_io' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_list_parallel.log 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.149 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo ec_io 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.150 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.151 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ list_parallel: /' 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=ec_io 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41791 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test ec_io on pid 41791' 2024-07-27T14:42:27.152 INFO:tasks.workunit.client.0.smithi087.stdout:test ec_io on pid 41791 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41791 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.168 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.169 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.170 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.171 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41663/exe 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.172 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.173 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.174 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s list 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' list' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.175 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.176 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.177 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.178 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.179 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.180 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo list 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=list 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41865 2024-07-27T14:42:27.181 INFO:tasks.workunit.client.0.smithi087.stdout:test list on pid 41865 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test list on pid 41865' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41865 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.190 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s ec_list 2024-07-27T14:42:27.191 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' ec_list' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.192 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo ec_list 2024-07-27T14:42:27.193 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41696/exe 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.194 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.195 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ cls: /' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_cls.log 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_cls 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.196 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.197 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=ec_list 2024-07-27T14:42:27.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41911 2024-07-27T14:42:27.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test ec_list on pid 41911' 2024-07-27T14:42:27.198 INFO:tasks.workunit.client.0.smithi087.stdout:test ec_list on pid 41911 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41911 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.205 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.206 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41702/exe 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.207 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s misc 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' misc' 2024-07-27T14:42:27.208 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo misc 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.209 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.210 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.211 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=misc 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41936 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test misc on pid 41936' 2024-07-27T14:42:27.212 INFO:tasks.workunit.client.0.smithi087.stdout:test misc on pid 41936 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=41936 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_cmd 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_cmd.log 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ cmd: /' 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.228 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.229 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s pool 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.230 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' pool' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.231 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.232 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41754/exe 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.233 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.234 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.235 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41729/exe 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ io: /' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.237 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_io 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo pool 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_io.log 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.240 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_handler_error 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ handler_error: /' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.241 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_handler_error.log 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=pool 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42000 2024-07-27T14:42:27.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test pool on pid 42000' 2024-07-27T14:42:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:test pool on pid 42000 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=42000 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.251 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.253 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s read_operations 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' read_operations' 2024-07-27T14:42:27.254 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo read_operations 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.255 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.256 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.257 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.258 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.259 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=read_operations 2024-07-27T14:42:27.260 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42026 2024-07-27T14:42:27.260 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test read_operations on pid 42026' 2024-07-27T14:42:27.260 INFO:tasks.workunit.client.0.smithi087.stdout:test read_operations on pid 42026 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=42026 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s snapshots 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.264 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' snapshots' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.265 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo snapshots 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.266 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=snapshots 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.267 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.268 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.268 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.268 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.268 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42039 2024-07-27T14:42:27.268 INFO:tasks.workunit.client.0.smithi087.stdout:test snapshots on pid 42039 2024-07-27T14:42:27.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test snapshots on pid 42039' 2024-07-27T14:42:27.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=42039 2024-07-27T14:42:27.273 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.273 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s watch_notify 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' watch_notify' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41936/exe 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.274 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.275 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo watch_notify 2024-07-27T14:42:27.276 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_misc 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.277 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41911/exe 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ misc: /' 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=watch_notify 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42059 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test watch_notify on pid 42059' 2024-07-27T14:42:27.278 INFO:tasks.workunit.client.0.smithi087.stdout:test watch_notify on pid 42059 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=42059 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_misc.log 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.298 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_ec_list 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ ec_list: /' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.300 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.301 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s write_operations 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' write_operations' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.302 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.303 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41865/exe 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo write_operations 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.305 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.306 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.307 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_ec_list.log 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.308 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.309 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.310 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ list: /' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.311 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_list.log 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_list 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/41791/exe 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.312 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.313 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.314 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.315 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_ec_io.log 2024-07-27T14:42:27.316 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_ec_io 2024-07-27T14:42:27.316 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=write_operations 2024-07-27T14:42:27.316 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42108 2024-07-27T14:42:27.316 INFO:tasks.workunit.client.0.smithi087.stdout:test write_operations on pid 42108 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test write_operations on pid 42108' 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=42108 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T14:42:27.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41091 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41091 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.344 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.345 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.346 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.347 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.348 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/42026/exe 2024-07-27T14:42:27.349 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ ec_io: /' 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.350 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -f /etc/bashrc ']' 2024-07-27T14:42:27.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /etc/bashrc 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_read_operations 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -z '' ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:++ BASHRCSOURCED=Y 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:++ shopt -q login_shell 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.352 INFO:tasks.workunit.client.0.smithi087.stderr:+++ umask 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' 0022 -eq 0 ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ SHELL=/bin/bash 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-27T14:42:27.353 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ read_operations: /' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.354 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'grep=grep --color=auto' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-27T14:42:27.355 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorls.sh 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' '!' -t 0 ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:+++ return 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export S_COLORS=auto 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:+++ S_COLORS=auto 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-27T14:42:27.356 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_read_operations.log 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.357 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-27T14:42:27.358 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/42000/exe 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.359 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/42059/exe 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.360 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.361 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ pool: /' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ watch_notify: /' 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_pool 2024-07-27T14:42:27.362 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.363 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_pool.log 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_watch_notify 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/42039/exe 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.364 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.365 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/gawk.sh 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/lang.sh 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_snapshots.log 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_snapshots 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/locale 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ snapshots: /' 2024-07-27T14:42:27.366 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:++++ export LANG=en_US.UTF-8 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:++++ LANG=en_US.UTF-8 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ LANG=en_US.UTF-8 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ unset LANG_backup config 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n '' ']' 2024-07-27T14:42:27.367 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' dumb = linux ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/less.sh 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ for i in /etc/profile.d/*.sh 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' '' ']' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++ . /etc/profile.d/which2.sh 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++++ readlink /proc/42108/exe 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:++++ basename /usr/bin/bash 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_declare='declare -f' 2024-07-27T14:42:27.368 INFO:tasks.workunit.client.0.smithi087.stderr:+++ which_opt=-f 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export which_declare 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+++ export -f which 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset i 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:++ unset -f pathmunge 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ export PATH 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ unset rc 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_neorados_write_operations 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_neorados_write_operations.log 2024-07-27T14:42:27.369 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ write_operations: /' 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout:88640 10 monclient: get_auth_request con 0x7f28d406ba60 auth_method 0 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28f1088640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f28d406ba60 0x7f28d406df20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28e400b880 tx=0x7f28e4001470 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 10 client.4417.objecter ms_handle_connect 0x7f28d406ba60 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5238+0+0 (secure 0 0 0) 0x7f28e8098350 con 0x560458750d40 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(25..25 src has 1..25) 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [25,25] > 0 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding full epoch 25 2024-07-27T14:42:28.416 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28f3e3a780 10 librados: wait_for_osdmap done waiting 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28f3e3a780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.828+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28e80527c0 con 0x560458750d40 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f28e8096550 con 0x560458750d40 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28e27fc640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28e2ffd640 10 client.4417.objecter _get_latest_version latest 25, have it 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28f3e3a780 10 client.4417.objecter create_pool name=ceph_test_rados_api_asio 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28f3e3a780 10 client.4417.objecter pool_op_submit 1 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28f3e3a780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:26.843+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x56045863e750 con 0x560458750d40 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v26) ==== 43+0+0 (secure 0 0 0) 0x7f28e8014f10 con 0x560458750d40 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 pool_op_reply(tid 1 (0) Success v26) 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 client.4417.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v26) 2024-07-27T14:42:28.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 client.4417.objecter have request 1 at 0x560458686f10 Op: create 2024-07-27T14:42:28.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 20 client.4417.objecter waiting for client to reach epoch 26 before calling back 2024-07-27T14:42:28.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:42:28.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 monclient: _renew_subs 2024-07-27T14:42:28.423 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=26}) -- 0x7f28d4071740 con 0x560458750d40 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.415+0000 7f28e27fc640 10 client.4417.objecter done 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 8 ==== osd_map(26..26 src has 1..26) ==== 1748+0+0 (secure 0 0 0) 0x7f28e80985b0 con 0x560458750d40 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(26..26 src has 1..26) 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [26,26] > 25 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 26 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 10 librados: set snap read head -> 1 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 10 librados: writefull oid=exist nspace= 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f45e0 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 26 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:28.424 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [writefull 0~5 in=5b] tid 2 osd.2 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 2 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 15 client.4417.objecter _send_op 2 to 4.1 on osd.2 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:2 4.1 4:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) -- 0x560458732860 con 0x5604587f4a90 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.419+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.420+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:42:28.425 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.420+0000 7f28f1889640 10 monclient: get_auth_request con 0x5604587f4a90 auth_method 0 2024-07-27T14:42:31.824 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:34.691 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:31.950 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:34.817 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.005 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:34.857 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.009 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:34.862 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.093 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:34.945 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.114 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:34.981 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.157 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:35.024 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.243 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:35.109 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:32.303 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:35.169 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.423+0000 7f28e27fc640 10 client.4417.objecter ms_handle_connect 0x5604587f4a90 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:28.423+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:29.765+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:29.765+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:29.765+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:29.765+0000 7f28e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:30.765+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:30.765+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:30.765+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.757 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:31.765+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:31.765+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:31.765+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:31.800+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:32.765+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:32.765+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:32.765+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:33.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:33.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:33.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:34.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:34.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:34.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 1 ==== osd_map(27..28 src has 1..28) ==== 21199+0+0 (crc 0 0 0) 0x7f28ec00e040 con 0x5604587f4a90 2024-07-27T14:42:35.758 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(27..28 src has 1..28) 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [27,28] > 26 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 27 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 10 client.4417.objecter checking op 2 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 27 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 28 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 10 client.4417.objecter checking op 2 2024-07-27T14:42:35.759 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 28 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:39.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter _calc_tar api_service: [==========] Running 4 tests from 1 test suite. 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] Global test environment set-up. 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] 4 tests from LibRadosService 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5911 ms) 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.RegisterLate (588 ms) 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_WARN 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 13 pool(s) do not have an application enabled 2024-07-27T14:42:39.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 84s), 8 in (since 2m) 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 33 pools, 1001 pgs 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 4 objects, 449 KiB 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 98.402% pgs unknown 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 0.699% pgs not active 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 985 unknown 2024-07-27T14:42:39.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 9 active+clean 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 7 creating+activating 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_WARN 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 15 pool(s) do not have an application enabled 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 87s), 8 in (since 2m) 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-07-27T14:42:40.000 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 34 pools, 961 pgs 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 4 objects, 449 KiB 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 81.894% pgs unknown 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 17.066% pgs not active 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 787 unknown 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 136 creating+peering 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 28 creating+activating 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 10 active+clean 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T14:42:40.001 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_WARN 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 20 pool(s) do not have an application enabled 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 89s), 8 in (since 2m) 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 41 pools, 1209 pgs 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 5 objects, 449 KiB 2024-07-27T14:42:40.002 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-27T14:42:40.996 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 43.672% pgs unkget target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:40.996 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:40.996 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.756+0000 7f28e27fc640 20 client.4417.objecter 2 4.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-07-27T14:42:40.996 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.996 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:35.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:36.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:36.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:36.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:36.800+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:37.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:37.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:37.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:38.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:38.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:38.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:39.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:39.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.997 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:39.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:39.766+0000 7f28e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.766+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.766+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.766+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 2 ==== osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 2 ondisk uv 1 in 4.1 attempt 0 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 2 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 15 client.4417.objecter _finish_op 2 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 2 2024-07-27T14:42:40.998 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:40.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f3e3a780 10 librados: Objecter returned from writefull r=0 2024-07-27T14:42:40.999 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.995+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 28 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~256] tid 3 osd.2 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 3 2024-07-27T14:42:41.438 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 15 client.4417.objecter _send_op 3 to 4.1 on osd.2 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:3 4.1 4:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) -- 0x5604587f3a80 con 0x5604587f4a90 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f3e70 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 28 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~256] tid 4 osd.2 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 4 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 15 client.4417.objecter _send_op 4 to 4.1 on osd.2 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:4 4.1 4:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) -- 0x5604587f4320 con 0x5604587f4a90 2024-07-27T14:42:41.439 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:40.996+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 3 ==== osd_map(29..32 src has 1..34) ==== 8658+0+0 (crc 0 0 0) 0x7f28ec00d740 con 0x5604587f4a90 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(29..32 src has 1..34) 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [29,32] > 28 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 29 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 3 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 29 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 29 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 30 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 3 2024-07-27T14:42:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 30 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 30 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 31 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 3 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 31 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 31 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 32 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 3 2024-07-27T14:42:41.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 32 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 32 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:41.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter 3 4.d3a59201 osd.2 exist [read 0~256] 2024-07-27T14:42:41.943 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.437+0000 7f28e27fc640 20 client.4417.objecter 4 4.a7bbaa1 osd.2 noexist [read 0~256] 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 3 ondisk uv 1 in 4.1 attempt 0 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 3 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 15 client.4417.objecter _finish_op 3 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 3 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.574+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:41.944 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.801+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 5 ==== osd_map(33..34 src has 1..34) ==== 3908+0+0 (crc 0 0 0) 0x7f28ec00d940 con 0x5604587f4a90 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(33..34 src has 1..34) 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 6 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [33,34] > 32 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 33 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 33 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 34 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 10 client.4417.objecter checking op 4 2024-07-27T14:42:41.945 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 34 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28e27fc640 20 client.4417.objecter 4 4.a7bbaa1 osd.2 noexist [read 0~256] 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 4 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 4 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 15 client.4417.objecter _finish_op 4 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 4 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (948 ms) 2024-07-27T14:42:41.946 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f3e70 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~256] tid 5 osd.2 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 5 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 15 client.4417.objecter _send_op 5 to 4.1 on osd.2 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:5 4.1 4:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x5604587f46c0 con 0x5604587f4a90 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:41.947 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:41.948 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:41.948 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:41.948 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~256] tid 6 osd.2 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 6 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 15 client.4417.objecter _send_op 6 to 4.1 on osd.2 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:6 4.1 4:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x560458732c30 con 0x5604587f4a90 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:41.943+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 5 ondisk uv 1 in 4.1 attempt 0 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 5 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter _finish_op 5 2024-07-27T14:42:42.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 5 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 6 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 6 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter _finish_op 6 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 6 2024-07-27T14:42:42.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (77 ms) 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:42.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~256] tid 7 osd.2 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 7 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 15 client.4417.objecter _send_op 7 to 4.1 on osd.2 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:7 4.1 4:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x5604587f3e40 con 0x5604587f4a90 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587337e0 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~256] tid 8 osd.2 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 8 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 15 client.4417.objecter _send_op 8 to 4.1 on osd.2 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:8 4.1 4:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) -- 0x560458733c90 con 0x5604587f4a90 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.020+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 7 ondisk uv 1 in 4.1 attempt 0 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 7 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter _finish_op 7 2024-07-27T14:42:42.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 7 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 8 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 8 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter _finish_op 8 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 8 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:42.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (188 ms) 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x5604585d5700 write_seq 1 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587337e0 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [write 0~5 in=5b] tid 9 osd.2 2024-07-27T14:42:42.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:49.271 INFO:tasks.workunit.client.0.smithi087.stdout: pool: Running main() from gmock_main.cc 2024-07-27T14:42:49.271 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-07-27T14:42:49.271 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [----------] Global test environment set-up. 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolList (4666 ms) 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4261 ms) 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2745 ms) 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5165 ms) 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2554 ms) 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-07-27T14:42:49.272 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2731 ms) 2024-07-27T14:42:49.273 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [----------] 6 tests from NeoRadosPools (22122 ms total) 2024-07-27T14:42:49.273 INFO:tasks.workunit.client.0.smithi087.stdout: pool: 2024-07-27T14:42:49.273 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [----------] Global test environment tear-down 2024-07-27T14:42:49.273 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [==========] 6 tests from 1 test suite ran. (22122 ms total) 2024-07-27T14:42:49.273 INFO:tasks.workunit.client.0.smithi087.stdout: pool: [ PASSED ] 6 tests. 2024-07-27T14:42:51.768 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 9 2024-07-27T14:42:51.768 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 15 client.4417.objecter _send_op 9 to 4.1 on osd.2 2024-07-27T14:42:51.768 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:9 4.1 4:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) -- 0x5604587f35d0 con 0x5604587f4a90 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.208+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.209+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.209+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:42.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:43.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:43.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:43.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:44.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:44.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:44.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:45.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:45.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:45.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:46.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:46.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:46.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:46.802+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:47.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:47.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:47.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:48.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:48.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:48.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:49.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:49.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:49.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:51.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:49.767+0000 7f28e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:42:51.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:50.767+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:51.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:50.767+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:51.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:50.767+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:52.127 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:54.993 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.227 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:55.093 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.294 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:55.146 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.300 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:55.152 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.305 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:55.172 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.307 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:55.174 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.520 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:02:55.372 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.540 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:55.407 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.699 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:55.565 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:42:52.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:51.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:52.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:51.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:52.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:51.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:52.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:51.803+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:52.868 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (10659 ms) 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.866+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 11 ==== osd_op_reply(9 exist [write 0~5] v34'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(9 exist [write 0~5] v34'2 uv2 ondisk = 0) 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 9 ondisk uv 2 in 4.1 attempt 0 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 9 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 15 client.4417.objecter _finish_op 9 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:52.869 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 9 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f1889640 20 librados: complete_aio_write 0x5604585d5700 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x5604585de020 write_seq 2 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 34 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [write 0~5 in=5b] tid 10 osd.2 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 10 2024-07-27T14:42:52.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 15 client.4417.objecter _send_op 10 to 4.1 on osd.2 2024-07-27T14:42:52.932 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:10 4.1 4:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) -- 0x5604587337c0 con 0x5604587f4a90 2024-07-27T14:42:52.932 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.867+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 12 ==== osd_map(35..41 src has 1..41) ==== 5168+0+0 (crc 0 0 0) 0x7f28ec00dbe0 con 0x5604587f4a90 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(35..41 src has 1..41) 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [35,41] > 34 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 35 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 35 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 36 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 36 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.933 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 37 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 37 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 38 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 38 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 39 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 39 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:52.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 40 2024-07-27T14:42:52.935 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:54.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 40 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:54.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 41 2024-07-27T14:42:54.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 10 client.4417.objecter checking op 10 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 41 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:52.932+0000 7f28e27fc640 20 client.4417.objecter 10 4.d3a59201 osd.2 exist [write 0~5 in=5b] 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:53.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:53.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:53.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 13 ==== osd_op_reply(10 exist [write 0~5] v41'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(10 exist [write 0~5] v41'3 uv3 ondisk = 0) 2024-07-27T14:42:54.102 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 10 ondisk uv 3 in 4.1 attempt 0 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 10 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 15 client.4417.objecter _finish_op 10 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 10 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f1889640 20 librados: complete_aio_write 0x5604585de020 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1233 ms) 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x5604587338b0 write_seq 3 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:54.103 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 41 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:54.104 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.104 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:54.104 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:54.811 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [write 0~5 in=5b] tid 11 osd.2 2024-07-27T14:42:54.811 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:54.811 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 11 2024-07-27T14:42:54.811 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 15 client.4417.objecter _send_op 11 to 4.1 on osd.2 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:11 4.1 4:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) -- 0x560458737430 con 0x5604587f4a90 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.100+0000 7f28f3e3a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 14 ==== osd_map(42..42 src has 1..42) ==== 675+0+0 (crc 0 0 0) 0x7f28ec00dbe0 con 0x5604587f4a90 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(42..42 src has 1..42) 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [42,42] > 41 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 42 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 10 client.4417.objecter checking op 11 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.207+0000 7f28e27fc640 20 client.4417.objecter 11 4.d3a59201 osd.2 exist [write 0~5 in=5b] 2024-07-27T14:42:54.812 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 15 ==== osd_op_reply(11 exist [write 0~5] v42'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(11 exist [write 0~5] v42'4 uv4 ondisk = 0) 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (711 ms) 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 11 ondisk uv 4 in 4.1 attempt 0 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 11 2024-07-27T14:42:54.813 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 15 client.4417.objecter _finish_op 11 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 11 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f1889640 20 librados: complete_aio_write 0x5604587338b0 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587f35d0 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.906 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~0] tid 12 osd.2 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 12 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 15 client.4417.objecter _send_op 12 to 4.1 on osd.2 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:12 4.1 4:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x560458736dc0 con 0x5604587f4a90 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:54.907 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458737160 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~0] tid 13 osd.2 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 13 2024-07-27T14:42:54.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 15 client.4417.objecter _send_op 13 to 4.1 on osd.2 2024-07-27T14:42:54.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:13 4.1 4:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x560458734030 con 0x5604587f4a90 2024-07-27T14:42:54.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:54.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.811+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:54.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:54.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-27T14:42:54.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:54.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 12 ondisk uv 4 in 4.1 attempt 0 2024-07-27T14:42:54.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 12 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter _finish_op 12 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 12 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:54.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:54.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:54.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:54.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 13 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:54.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:54.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 13 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter _finish_op 13 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 13 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:54.913 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (94 ms) 2024-07-27T14:42:54.914 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-07-27T14:42:54.914 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458736dc0 2024-07-27T14:42:54.914 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:54.914 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:54.914 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:54.915 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:54.915 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~0] tid 14 osd.2 2024-07-27T14:42:54.915 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:55.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 14 2024-07-27T14:42:55.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 15 client.4417.objecter _send_op 14 to 4.1 on osd.2 2024-07-27T14:42:55.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:14 4.1 4:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x5604587f36e0 con 0x5604587f4a90 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458734030 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~0] tid 15 osd.2 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 15 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 15 client.4417.objecter _send_op 15 to 4.1 on osd.2 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:15 4.1 4:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x560458737450 con 0x5604587f4a90 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:54.905+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.068+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:55.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 14 ondisk uv 4 in 4.1 attempt 0 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 14 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter _finish_op 14 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 14 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 15 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:55.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 15 2024-07-27T14:42:55.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter _finish_op 15 2024-07-27T14:42:55.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 15 2024-07-27T14:42:55.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:55.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (164 ms) 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458736dc0 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:55.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [read 0~0] tid 16 osd.2 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 16 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 15 client.4417.objecter _send_op 16 to 4.1 on osd.2 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:16 4.1 4:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x560458737410 con 0x5604587f4a90 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x5604587346a0 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base noexist @4 precalc_pgid 0 pgid 0.0 is_read 2024-07-27T14:42:55.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target noexist @4 -> pgid 4.a7bbaa1 2024-07-27T14:42:55.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.a7bbaa1 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid noexist '@4' '@4' [read 0~0] tid 17 osd.2 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 17 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 15 client.4417.objecter _send_op 17 to 4.1 on osd.2 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:17 4.1 4:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) -- 0x560458734b50 con 0x5604587f4a90 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.069+0000 7f28f3e3a780 5 client.4417.objecter 2 in flight 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:55.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 16 ondisk uv 4 in 4.1 attempt 0 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 5 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 16 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 15 client.4417.objecter _finish_op 16 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 5 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 16 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.151+0000 7f28f1889640 5 client.4417.objecter 1 in flight 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 17 ondisk uv 0 in 4.1 attempt 0 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 10 client.4417.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 17 2024-07-27T14:42:55.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 15 client.4417.objecter _finish_op 17 2024-07-27T14:42:55.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 17 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (97 ms) 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x560458737300 write_seq 4 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458736dc0 2024-07-27T14:42:55.965 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [writefull 0~5 in=5b] tid 18 osd.2 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 18 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 15 client.4417.objecter _send_op 18 to 4.1 on osd.2 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:18 4.1 4:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) -- 0x560458737410 con 0x5604587f4a90 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.166+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:55.966 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 22 ==== osd_op_reply(18 exist [writefull 0~5] v42'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(18 exist [writefull 0~5] v42'5 uv5 ondisk = 0) 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 18 ondisk uv 5 in 4.1 attempt 0 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 18 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 15 client.4417.objecter _finish_op 18 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 18 2024-07-27T14:42:55.967 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.964+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:57.339 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f1889640 20 librados: complete_aio_write 0x560458737300 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (799 ms) 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x5604587378b0 write_seq 5 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458736dc0 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [writefull 0~5 in=5b] tid 19 osd.2 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 19 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 15 client.4417.objecter _send_op 19 to 4.1 on osd.2 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:19 4.1 4:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) -- 0x560458737410 con 0x5604587f4a90 2024-07-27T14:42:57.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:55.965+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:56.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:56.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:56.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:56.804+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 23 ==== osd_op_reply(19 exist [writefull 0~5] v42'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x5604587f4a90 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(19 exist [writefull 0~5] v42'6 uv6 ondisk = 0) 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 19 ondisk uv 6 in 4.1 attempt 0 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 19 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 15 client.4417.objecter _finish_op 19 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:57.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 19 2024-07-27T14:42:57.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:42:57.478 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f1889640 20 librados: complete_aio_write 0x5604587378b0 2024-07-27T14:42:57.478 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1374 ms) 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 librados: queue_aio_write 0x5604586020b0 completion 0x560458737410 write_seq 6 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 10 client.4417.objecter _op_submit op 0x560458736dc0 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 client.4417.objecter _calc_target epoch 42 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 10 client.4417.objecter _calc_target raw pgid 4.d3a59201 -> actual 4.1 acting [2,3] primary 2 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 client.4417.objecter _get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 10 client.4417.objecter _op_submit oid exist '@4' '@4' [writefull 0~5 in=5b] tid 20 osd.2 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 client.4417.objecter get_session s=0x5604586c9370 osd=2 3 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 15 client.4417.objecter _session_op_assign 2 20 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 15 client.4417.objecter _send_op 20 to 4.1 on osd.2 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:20 4.1 4:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42) -- 0x5604587380c0 con 0x5604587f4a90 2024-07-27T14:42:57.479 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.339+0000 7f28f3e3a780 5 client.4417.objecter 1 in flight 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 msgr2=0x56045874bd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 msgr2=0x56045874bd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 client.4417.objecter ms_handle_reset 0x5604587f4a90 session 0x5604586c9370 osd.2 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 10 client.4417.objecter reopen_session osd.2 session, addr now [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] 2024-07-27T14:42:57.480 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 msgr2=0x56045874bd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T14:42:57.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x5604587f4a90 0x56045874bd20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:42:57.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f28d4089c00 0x7f28d4098910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 15 client.4417.objecter _send_op 20 to 4.1 on osd.2 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- osd_op(unknown.0.0:20 4.1 4:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e42) -- 0x7f28d407b1b0 con 0x7f28d4089c00 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 10 monclient: _renew_subs 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.475+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=43}) -- 0x7f28d4089fe0 con 0x560458750d40 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== mon.1 v2:172.21.15.112:3300/0 9 ==== osd_map(43..46 src has 1..46) ==== 1456+0+0 (secure 0 0 0) 0x7f28e809a240 con 0x560458750d40 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(43..46 src has 1..46) 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [43,46] > 42 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 43 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 43 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:57.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 44 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 44 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 45 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 45 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 46 2024-07-27T14:42:57.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:42:58.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 46 api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] Global test environment set-up. 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5387 ms) 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (227 ms) 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20173 ms) 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-07-27T14:42:58.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: attempt 0 of 20 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: failed to deregister: 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: { 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "epoch": 12, 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "modified": "2024-07-27T14:42:53.765962+0000", 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "services": { 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "laundry": { 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "daemons": { 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "summary": "", 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "close-test-pid41591": { 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_epoch": 12, 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_stamp": "2024-07-27T14:42:53.006725+0000", 2024-07-27T14:42:58.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "gid": 4786, 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "addr": "172.21.15.87:0/614637332", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "metadata": { 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "arch": "x86_64", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_release": "squid", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-3573-g57fd7103 (57fd7103c78964675e1941b32d58041fb7c15af7) squid (rc)", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_version_short": "19.0.0-3573-g57fd7103", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro": "centos", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro_version": "9", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "foo": "bar", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "hostname": "smithi087", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-07-27T14:42:58.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "mem_swap_kb": "0", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "mem_total_kb": "32487260", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "os": "Linux", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "this": "that" 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: }, 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "task_status": {} 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: }, 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "pid41579": { 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_epoch": 11, 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_stamp": "2024-07-27T14:42:40.123173+0000", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "gid": 4753, 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "addr": "172.21.15.87:0/313928713", 2024-07-27T14:42:58.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "metadata": { 2024-07-27T14:42:58.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "arch": "x86_64", 2024-07-27T14:42:58.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_release": "squid", 2024-07-27T14:42:58.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-3573-g57fd7103 (57fd7103c78964675e1941b32d58041fb7c15af7) squid (rc)", 2024-07-27T14:42:58.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "ceph_version_short": "19.0.0-3573-g57fd7103", 2024-07-27T14:42:58.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-07-27T14:43:00.099 INFO:tasks.workunit.client.0.smithi087.stdout: nown 2024-07-27T14:43:00.099 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 53.763% pgs not active 2024-07-27T14:43:00.099 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 617 creating+peering 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 528 unknown 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 33 creating+activating 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 31 active+clean 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: io: 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: client: 209 B/s wr, 0 op/s rd, 0 op/s wr 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6621 ms) 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.Status 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.Status (20059 ms) 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] 4 tests from LibRadosService (33179 ms total) 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-07-27T14:43:00.100 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] Global test environment tear-down 2024-07-27T14:43:00.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [==========] 4 tests from 1 test suite ran. (33179 ms total) 2024-07-27T14:43:00.101 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ PASSED ] 4 tests. 2024-07-27T14:43:01.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:43:01.111+0000 ea5d640 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4552.0:2 28.6 28:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:01.245 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:01.146+0000 ea5d640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:02.203 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:02.191+0000 ea5d640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:02.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:43:02.381+0000 ea5d640 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4552.0:2 28.6 28:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:03.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro": "centos", 2024-07-27T14:43:03.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-07-27T14:43:03.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "distro_version": "9", 2024-07-27T14:43:03.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "foo": "bar", 2024-07-27T14:43:03.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "hostname": "smithi087", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "mem_swap_kb": "0", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "mem_total_kb": "32487260", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "os": "Linux", 2024-07-27T14:43:03.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "this": "that" 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: }, 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "task_status": {} 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: }, 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "osd": { 2024-07-27T14:43:03.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "daemons": { 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "summary": "", 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "0": { 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_epoch": 0, 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "start_stamp": "0.000000", 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "gid": 0, 2024-07-27T14:43:03.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "addr": "(unrecognized address family 0)/0", 2024-07-27T14:43:03.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "metadata": {}, 2024-07-27T14:43:03.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: "task_status": {} 2024-07-27T14:43:03.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: } 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: attempt 1 of 20 2024-07-27T14:43:03.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10596 ms) 2024-07-27T14:43:03.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36384 ms total) 2024-07-27T14:43:03.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: 2024-07-27T14:43:03.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] Global test environment tear-down 2024-07-27T14:43:03.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36384 ms total) 2024-07-27T14:43:03.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-07-27T14:43:03.512 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:03.421+0000 ea5d640 -1 osd.5 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:04.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:04.319+0000 12a65640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:04.606 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:04.580+0000 ea5d640 -1 osd.7 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:04.627 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:04.608+0000 ea5d640 -1 osd.5 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.542+0000 7f28e27fc640 20 client.4417.objecter 20 4.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:57.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:58.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.095 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:58.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:58.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:59.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:59.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:59.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:42:59.768+0000 7f28e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:00.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:00.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:00.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:01.768+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:01.768+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:01.768+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:01.805+0000 7f28f32ec640 10 client.4417.objecter tick 2024-07-27T14:43:05.096 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:02.769+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:02.769+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:02.769+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:03.769+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:03.769+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:03.769+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:04.769+0000 7f28e17fa640 10 monclient: tick 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:04.769+0000 7f28e17fa640 10 monclient: _check_auth_tickets 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:04.769+0000 7f28e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:04.975+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f28d4089c00 0x7f28d4098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:04.992+0000 7f28f1889640 10 monclient: get_auth_request con 0x7f28d4089c00 auth_method 0 2024-07-27T14:43:05.097 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.094+0000 7f28e27fc640 10 client.4417.objecter ms_handle_connect 0x7f28d4089c00 2024-07-27T14:43:05.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.094+0000 7f28f1889640 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f28d4089c00 0x7f28d4098910 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T14:43:05.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 1 ==== osd_map(43..49 src has 1..49) ==== 3695+0+0 (crc 0 0 0) 0x7f28ec02b460 con 0x7f28d4089c00 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 <== osd.2 v2:172.21.15.87:6820/357294290 2 ==== osd_op_reply(20 exist [writefull 0~5] v44'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f28ec00d4b0 con 0x7f28d4089c00 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_map(43..49 src has 1..49) 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 10 client.4417.objecter ms_dispatch 0x560458632e00 osd_op_reply(20 exist [writefull 0~5] v44'7 uv7 ondisk = 0) 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 10 client.4417.objecter in handle_osd_op_reply 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map got epochs [43,49] > 46 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 47 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 47 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 48 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:43:05.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 48 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 3 client.4417.objecter handle_osd_map decoding incremental epoch 49 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 10 client.4417.objecter checking op 20 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target epoch 49 base exist @4 precalc_pgid 0 pgid 0.0 is_write 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter _calc_target target exist @4 -> pgid 4.d3a59201 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28e27fc640 20 client.4417.objecter 20 4.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 7 client.4417.objecter handle_osd_op_reply 20 ondisk uv 7 in 4.1 attempt 1 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 10 client.4417.objecter op 0 rval 0 len 0 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 15 client.4417.objecter handle_osd_op_reply completed tid 20 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 15 client.4417.objecter _finish_op 20 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 4 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 15 client.4417.objecter _session_op_remove 2 20 2024-07-27T14:43:05.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 5 client.4417.objecter 0 in flight 2024-07-27T14:43:05.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 20 librados: complete_aio_write 0x560458737410 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (7827 ms) 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 10 librados: watch_flush enter 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 10 librados: watch_flush exit 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 10 client.4417.objecter close_session for osd.2 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f28d4089c00 msgr2=0x7f28d4098910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f28d4089c00 0x7f28d4098910 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 20 client.4417.objecter put_session s=0x5604586c9370 osd=2 1 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 20 client.4417.objecter shutdown clearing up homeless session... 2024-07-27T14:43:05.170 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 10 client.4417.objecter successfully canceled tick 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f28d406ba60 msgr2=0x7f28d406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f28d406ba60 0x7f28d406df20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28e400b880 tx=0x7f28e4001470 comp rx=0 tx=0).stop 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 10 monclient: shutdown 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 msgr2=0x5604587f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 --2- 172.21.15.87:0/403720986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560458750d40 0x5604587f60f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f28e8055040 tx=0x7f28dc003110 comp rx=0 tx=0).stop 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f1889640 1 -- 172.21.15.87:0/403720986 reap_dead start 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 shutdown_connections 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 >> 172.21.15.87:0/403720986 conn(0x5604586c8e50 msgr2=0x56045874e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 shutdown_connections 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 -- 172.21.15.87:0/403720986 wait complete. 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.166+0000 7f28f3e3a780 1 librados: shutdown 2024-07-27T14:43:05.171 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] 12 tests from AsioRados (24171 ms total) 2024-07-27T14:43:05.172 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-07-27T14:43:05.172 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] Global test environment tear-down 2024-07-27T14:43:05.172 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (38434 ms total) 2024-07-27T14:43:05.172 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ PASSED ] 12 tests. 2024-07-27T14:43:05.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:05.388+0000 12a65640 -1 osd.2 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:05.595 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:05.584+0000 ea5d640 -1 osd.7 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:05.948 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:05.850+0000 ea5d640 -1 osd.5 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:06.784 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:06.637+0000 ea5d640 -1 osd.7 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:07.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:07.078+0000 12a65640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:07.493 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:07.469+0000 12a65640 -1 osd.6 51 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:08.003 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:07.904+0000 ea5d640 -1 osd.7 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:08.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:08.251+0000 12a65640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:08.982 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:08.953+0000 ea5d640 -1 osd.7 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:09.107 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:09.064+0000 12a65640 -1 osd.6 52 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:09.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:09.414+0000 12a65640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:09.961 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:09.957+0000 ea5d640 -1 osd.7 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:10.342 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:10.331+0000 12a65640 -1 osd.6 54 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:10.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:10.443+0000 12a65640 -1 osd.2 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:10.753 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:10.727+0000 ea5d640 -1 osd.5 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:11.525 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:11.511+0000 12a65640 -1 osd.6 54 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:11.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:11.511+0000 12a65640 -1 osd.2 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:11.631 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:11.517+0000 ea5d640 -1 osd.7 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:11.715 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:11.708+0000 ea5d640 -1 osd.5 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:12.408 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:15.274 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.416 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:15.283 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.418 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:15.285 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.422 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:15.288 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.465 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:15.317 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.479 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:15.331 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.635 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:15.488 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.683 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:12.671+0000 ea5d640 -1 osd.7 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:12.689 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:15.555 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:12.634+0000 12a65640 -1 osd.2 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:12.779 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:15.644 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:12.822 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:12.734+0000 12a65640 -1 osd.6 55 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:12.862 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:12.843+0000 ea5d640 -1 osd.5 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:14.014 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:13.935+0000 12a65640 -1 osd.6 56 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:14.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:14.044+0000 12a65640 -1 osd.2 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:14.079 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:13.933+0000 ea5d640 -1 osd.5 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:14.293 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:14.248+0000 ea5d640 -1 osd.7 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:15.294 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:15.200+0000 12a65640 -1 osd.6 58 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:15.336 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:15.243+0000 ea5d640 -1 osd.5 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:15.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:15.343+0000 12a65640 -1 osd.2 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4561.0:2 29.1 29:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:16.114 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:15.896+0000 ea5d640 -1 osd.7 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4513.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:16.199 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:16.195+0000 12a65640 -1 osd.6 58 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:16.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:16.700+0000 12a65640 -1 osd.2 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4561.0:2 29.1 29:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:17.025 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:16.899+0000 ea5d640 -1 osd.5 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:17.403 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:17.357+0000 12a65640 -1 osd.6 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:17.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:17.805+0000 12a65640 -1 osd.2 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4561.0:2 29.1 29:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:18.354 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:18.194+0000 ea5d640 -1 osd.5 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:18.692 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:18.669+0000 12a65640 -1 osd.6 60 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4445.0:2 34.1b 34:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:19.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:19.006+0000 12a65640 -1 osd.2 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:19.667 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:19.567+0000 ea5d640 -1 osd.5 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:20.046 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:19.964+0000 12a65640 -1 osd.6 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4447.0:2 13.12 13.76134b72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:20.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:20.070+0000 12a65640 -1 osd.2 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:21.077 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:20.953+0000 ea5d640 -1 osd.5 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:21.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:21.240+0000 12a65640 -1 osd.2 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:22.053 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:21.989+0000 12a65640 -1 osd.6 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:22.092 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:22.083+0000 ea5d640 -1 osd.5 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:22.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:22.805+0000 12a65640 -1 osd.2 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:23.410 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:23.291+0000 ea5d640 -1 osd.5 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:23.477 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:23.346+0000 ea5d640 -1 osd.3 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:23.567 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:23.497+0000 12a65640 -1 osd.6 64 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:24.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:24.136+0000 12a65640 -1 osd.2 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: Running main() from gmock_main.cc 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] Global test environment set-up. 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolList (6538 ms) 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4027 ms) 2024-07-27T14:43:24.336 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4043 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3898 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4162 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (12006 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6719 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6031 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10157 ms) 2024-07-27T14:43:24.337 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] 9 tests from LibRadosPools (57581 ms total) 2024-07-27T14:43:24.338 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: 2024-07-27T14:43:24.338 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] Global test environment tear-down 2024-07-27T14:43:24.338 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (57581 ms total) 2024-07-27T14:43:24.338 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ PASSED ] 9 tests. 2024-07-27T14:43:24.665 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:24.573+0000 ea5d640 -1 osd.3 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:24.937 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:24.925+0000 ea5d640 -1 osd.5 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:24.971 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:24.956+0000 12a65640 -1 osd.6 65 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:25.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:25.427+0000 12a65640 -1 osd.2 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:25.852 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:25.761+0000 ea5d640 -1 osd.3 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:26.353 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:26.327+0000 ea5d640 -1 osd.5 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:26.463 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:26.456+0000 12a65640 -1 osd.6 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:27.451 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:27.288+0000 ea5d640 -1 osd.3 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:27.731 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:27.726+0000 ea5d640 -1 osd.5 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:27.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:27.847+0000 12a65640 -1 osd.2 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:28.074 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:28.068+0000 12a65640 -1 osd.6 68 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] Global test environment set-up. 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (1 ms total) 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:42:26.955+0000 7f0658640780 -1 failed for service _ceph-mon._tcp 2024-07-27T14:43:28.136 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:42:26.955+0000 7f0658640780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (227 ms) 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5015 ms) 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5242 ms total) 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5071740 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a642b0 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eeb5a0 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5994820 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4d76660 2024-07-27T14:43:28.137 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510f3e0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510fe30 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2e8c0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f330 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2fd50 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f0a0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30ca0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2fb60 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31bf0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f309a0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f327b0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110de0 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31930 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111940 2024-07-27T14:43:28.138 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51123f0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110b10 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51132c0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112180 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda260 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edad70 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113030 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edbca0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f1b0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31810 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30ab0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda100 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda9e0 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ef40 2024-07-27T14:43:28.139 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e50690d0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edaaf0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edac00 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110c20 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f440 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111f00 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f590 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f6a0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f7b0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2f8c0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110ef0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111510 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51116c0 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112500 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112b60 2024-07-27T14:43:28.140 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112c70 2024-07-27T14:43:28.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112d80 2024-07-27T14:43:28.141 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111a50 2024-07-27T14:43:28.788 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:28.782+0000 ea5d640 -1 osd.3 68 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:28.866 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:28.845+0000 ea5d640 -1 osd.5 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:28.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:28.867+0000 12a65640 -1 osd.2 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:29.173 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:29.125+0000 12a65640 -1 osd.6 69 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:29.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:29.857+0000 12a65640 -1 osd.2 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:29.916 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:29.903+0000 ea5d640 -1 osd.5 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:29.931 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:29.832+0000 ea5d640 -1 osd.3 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:30.034 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:29.916+0000 ea5d640 -1 osd.4 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:30.331 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:43:30.279+0000 12a65640 -1 osd.6 70 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4447.0:2 13.12 13:4ed2c86e:test-rados-api-smithi087-41235-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:31.203 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:31.186+0000 ea5d640 -1 osd.5 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:31.386 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:31.296+0000 ea5d640 -1 osd.3 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:31.509 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:31.307+0000 ea5d640 -1 osd.4 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:31.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:31.568+0000 12a65640 -1 osd.2 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:32.317 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:32.311+0000 ea5d640 -1 osd.4 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:32.766 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:32.566+0000 ea5d640 -1 osd.3 71 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:32.792 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:32.720+0000 ea5d640 -1 osd.5 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:33.386 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:33.293+0000 ea5d640 -1 osd.4 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:33.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:33.407+0000 12a65640 -1 osd.2 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:33.449 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:36.316 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.464 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:36.330 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.466 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:36.319 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.469 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:36.322 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.478 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:36.345 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.485 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:36.351 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.663 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:36.515 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.758 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:36.625 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.771 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:36.638 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:33.845 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:33.838+0000 ea5d640 -1 osd.5 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:33.978 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:33.969+0000 ea5d640 -1 osd.3 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:34.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:34.446+0000 12a65640 -1 osd.2 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:34.615 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:34.588+0000 ea5d640 -1 osd.4 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:35.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:35.509+0000 12a65640 -1 osd.2 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:35.548 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:35.496+0000 ea5d640 -1 osd.3 73 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:36.078 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:36.067+0000 ea5d640 -1 osd.4 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:36.245 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:43:36.216+0000 ea5d640 -1 osd.5 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 16.0 16:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-07-27T14:43:36.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:36.586+0000 12a65640 -1 osd.2 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:36.743 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:43:36.699+0000 ea5d640 -1 osd.3 74 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4467.0:11 33.1e 33:7be96e85:test-rados-api-smithi087-41485-3::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e39) 2024-07-27T14:43:37.515 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:37.502+0000 ea5d640 -1 osd.4 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:37.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:43:37.544+0000 ea5d640 -1 osd.1 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4607.0:5 57.8 57.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e48) 2024-07-27T14:43:37.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:37.712+0000 12a65640 -1 osd.2 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:38.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:38.878+0000 12a65640 -1 osd.2 77 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:39.084 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:39.074+0000 ea5d640 -1 osd.4 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:40.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:40.164+0000 12a65640 -1 osd.2 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:40.643 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:40.630+0000 ea5d640 -1 osd.4 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:41.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:41.573+0000 12a65640 -1 osd.2 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:42.103 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:42.031+0000 ea5d640 -1 osd.4 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:43.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:43.068+0000 12a65640 -1 osd.2 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:43.652 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:43.537+0000 ea5d640 -1 osd.4 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:44.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:44.234+0000 12a65640 -1 osd.2 81 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19.75e2010f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:46.180 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:46.051+0000 ea5d640 -1 osd.7 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4831.0:2 71.6 71.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e59) 2024-07-27T14:43:46.249 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:46.242+0000 ea5d640 -1 osd.4 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:46.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:46.498+0000 12a65640 -1 osd.2 81 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:47.290 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:47.234+0000 ea5d640 -1 osd.4 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:47.446 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T14:43:47.432+0000 ea5d640 -1 osd.7 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4831.0:2 71.6 71:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59) 2024-07-27T14:43:47.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:47.916+0000 12a65640 -1 osd.2 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:48.620 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:48.452+0000 ea5d640 -1 osd.4 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:49.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:48.959+0000 12a65640 -1 osd.2 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:49.675 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:49.598+0000 ea5d640 -1 osd.4 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:50.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:50.218+0000 12a65640 -1 osd.2 84 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:51.271 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:51.191+0000 ea5d640 -1 osd.4 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:51.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:51.462+0000 12a65640 -1 osd.2 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:52.276 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:52.253+0000 ea5d640 -1 osd.4 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:52.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:52.488+0000 12a65640 -1 osd.2 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:53.281 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:56.134 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.286 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:56.153 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.290 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:56.156 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.291 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:56.158 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.295 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:56.162 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.297 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:56.149 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.453 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:03:56.305 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.457 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:56.324 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.486 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:56.352 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:43:53.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:53.602+0000 12a65640 -1 osd.2 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:54.066 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:54.059+0000 ea5d640 -1 osd.4 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:54.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:54.815+0000 12a65640 -1 osd.2 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:55.174 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:55.100+0000 ea5d640 -1 osd.4 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:56.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:56.005+0000 12a65640 -1 osd.2 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:56.577 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:56.559+0000 ea5d640 -1 osd.4 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:57.125 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: starte watch_notify: Running main() from gmock_main.cc 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [----------] Global test environment set-up. 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: handle_notify cookie 94262811824208 notify_id 214748364800 notifier_gid 4552 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (45901 ms) 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: Trying... 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: handle_notify cookie 94262823906624 notify_id 369367187456 notifier_gid 4831 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: Waiting for 3.000000000s 2024-07-27T14:43:57.126 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: Timed out. 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: Flushing... 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: Flushed... 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (44071 ms) 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (89972 ms total) 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [----------] Global test environment tear-down 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (89972 ms total) 2024-07-27T14:43:57.127 INFO:tasks.workunit.client.0.smithi087.stdout: watch_notify: [ PASSED ] 2 tests. 2024-07-27T14:43:57.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:57.158+0000 12a65640 -1 osd.2 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:57.774 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:57.737+0000 ea5d640 -1 osd.4 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:58.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:58.495+0000 12a65640 -1 osd.2 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:43:59.038 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:43:59.034+0000 ea5d640 -1 osd.4 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:43:59.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:43:59.689+0000 12a65640 -1 osd.2 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4462.0:2 19.f 19:f08047ae:test-rados-api-smithi087-41261-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-07-27T14:44:00.248 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:44:00.224+0000 ea5d640 -1 osd.4 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4416.0:3 7.18 7:18c494af:test-rados-api-smithi087-41086-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-07-27T14:44:01.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:01.467+0000 ea5d640 -1 osd.0 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:01.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:44:01.409+0000 12a65640 -1 osd.2 95 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4400.0:1 40.4s0 40:3e543e61:test-rados-api-smithi087-41444-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-07-27T14:44:02.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:44:02.471+0000 12a65640 -1 osd.2 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4400.0:1 40.4s0 40:3e543e61:test-rados-api-smithi087-41444-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-07-27T14:44:02.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:02.685+0000 ea5d640 -1 osd.0 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:03.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:03.854+0000 ea5d640 -1 osd.0 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:05.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:05.246+0000 ea5d640 -1 osd.0 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:06.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:06.329+0000 ea5d640 -1 osd.0 98 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:07.412 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T14:44:07.360+0000 ea5d640 -1 osd.5 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4602.0:2 74.1e 74:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) 2024-07-27T14:44:07.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:07.406+0000 ea5d640 -1 osd.0 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:07.711 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:44:07.579+0000 ea5d640 -1 osd.4 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4576.0:10 31.19 31.5c889059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-07-27T14:44:08.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:08.450+0000 ea5d640 -1 osd.0 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:08.779 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:44:08.648+0000 ea5d640 -1 osd.4 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4576.0:10 31.19 31.5c889059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e64) 2024-07-27T14:44:11.739 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:11.731+0000 ea5d640 -1 osd.3 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:12.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:11.925+0000 ea5d640 -1 osd.0 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4855.0:2 76.15 76:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:12.873 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:12.826+0000 ea5d640 -1 osd.3 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:14.063 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:14.057+0000 ea5d640 -1 osd.3 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:15.180 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:15.061+0000 ea5d640 -1 osd.3 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:16.154 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:16.139+0000 ea5d640 -1 osd.3 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:17.392 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:17.324+0000 ea5d640 -1 osd.3 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:18.963 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:18.906+0000 ea5d640 -1 osd.3 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:27.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:44:26.997+0000 ea5d640 -1 osd.0 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4885.0:2 83.0 83.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e93) 2024-07-27T14:44:33.841 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:36.693 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:33.843 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:36.695 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:33.851 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:36.717 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:33.890 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:36.756 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:33.942 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:36.809 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:33.977 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:36.844 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:34.067 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:36.919 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:34.211 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:37.078 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:34.291 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:34.136+0000 ea5d640 -1 osd.3 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:34.378 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:37.244 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:35.273 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:35.264+0000 ea5d640 -1 osd.3 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:36.427 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:36.369+0000 ea5d640 -1 osd.3 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:37.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:37.048+0000 ea5d640 -1 osd.1 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:37.697 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:37.528+0000 ea5d640 -1 osd.3 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:38.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:38.586+0000 ea5d640 -1 osd.1 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:38.681 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:38.637+0000 ea5d640 -1 osd.3 126 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:40.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:40.031+0000 ea5d640 -1 osd.1 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:40.506 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:40.486+0000 ea5d640 -1 osd.3 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:41.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:41.456+0000 ea5d640 -1 osd.1 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:42.384 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:42.348+0000 ea5d640 -1 osd.3 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:42.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:42.867+0000 ea5d640 -1 osd.1 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:43.733 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:43.528+0000 ea5d640 -1 osd.3 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:43.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:43.856+0000 ea5d640 -1 osd.1 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:44.640 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:44.600+0000 ea5d640 -1 osd.3 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:45.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:44:45.538+0000 ea5d640 -1 osd.1 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4488.0:5 17.2 17:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91) 2024-07-27T14:44:46.015 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:45.966+0000 ea5d640 -1 osd.3 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:47.110 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:47.025+0000 ea5d640 -1 osd.3 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:48.367 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-07-27T14:44:48.367 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-07-27T14:44:48.367 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (141406 ms) 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (141406 ms total) 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (141406 ms total) 2024-07-27T14:44:48.368 INFO:tasks.workunit.client.0.smithi087.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-07-27T14:44:48.449 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:48.373+0000 ea5d640 -1 osd.3 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:49.620 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:49.616+0000 ea5d640 -1 osd.3 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout:d 0x55f9e51117d0 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111b80 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111cd0 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30db0 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31410 2024-07-27T14:44:50.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31520 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ea50 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ed9d60 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ed9eb0 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ed9fe0 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30f40 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31050 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31180 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31290 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111020 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2eb60 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ec70 2024-07-27T14:44:50.150 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31a40 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ed80 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e50703a0 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e50704d0 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e50705e0 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31630 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e50706f0 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda410 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111130 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111240 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5111390 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a57220 2024-07-27T14:44:50.151 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a57390 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a574a0 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a575b0 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e59b4130 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e59b4280 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e59b43f0 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e59b4560 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112630 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112740 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5112850 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51129a0 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5068900 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5068a10 2024-07-27T14:44:50.152 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5068b20 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5068c70 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110900 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a54810 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a54920 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a54a30 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5a54b40 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2fe60 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30480 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30590 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f306a0 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f307f0 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ffd0 2024-07-27T14:44:50.153 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda520 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda630 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eda780 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f321b0 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f322e0 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f32430 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f32540 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f32650 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb7e0 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb930 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edba80 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e506e760 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e506e870 2024-07-27T14:44:50.154 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e506e980 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e506ead0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31d00 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f300e0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f301f0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f30340 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510ffa0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110450 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5110560 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51106b0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51100b0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51101c0 2024-07-27T14:44:50.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51102d0 2024-07-27T14:44:50.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51054f0 2024-07-27T14:44:50.803 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:50.780+0000 ea5d640 -1 osd.3 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:51.995 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:51.974+0000 ea5d640 -1 osd.3 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:53.148 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:53.079+0000 ea5d640 -1 osd.3 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:54.102 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:56.968 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.110 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:56.963 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.112 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:56.978 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.116 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:56.983 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.118 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:56.984 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.186 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:57.038 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.276 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:57.142 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.293 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:04:57.145 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.325 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:57.192 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:44:54.331 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:54.273+0000 ea5d640 -1 osd.3 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:55.373 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:55.308+0000 ea5d640 -1 osd.3 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:55.404 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:44:55.397+0000 12a65640 -1 osd.6 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4879.0:5 82.6 82:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) 2024-07-27T14:44:56.585 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:44:56.480+0000 12a65640 -1 osd.6 144 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4879.0:5 82.6 82:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) 2024-07-27T14:44:56.706 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:56.694+0000 ea5d640 -1 osd.3 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:57.468 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:44:57.462+0000 12a65640 -1 osd.6 144 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4879.0:5 82.6 82:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) 2024-07-27T14:44:58.117 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:58.054+0000 ea5d640 -1 osd.3 144 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:44:59.091 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:44:59.086+0000 12a65640 -1 osd.6 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4879.0:5 82.6 82:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) 2024-07-27T14:44:59.270 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:44:59.232+0000 ea5d640 -1 osd.3 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:00.289 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:00.275+0000 ea5d640 -1 osd.3 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:01.913 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:01.864+0000 ea5d640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:03.104 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:03.096+0000 ea5d640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:04.597 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:04.575+0000 ea5d640 -1 osd.3 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:05.662 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:05.647+0000 ea5d640 -1 osd.3 150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:06.931 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:06.811+0000 ea5d640 -1 osd.3 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:07.118 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T14:45:06.961+0000 ea5d640 -1 osd.4 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4909.0:2 88.15 88:ad551702:::test:head [call hello.record_hello in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) 2024-07-27T14:45:08.316 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:08.187+0000 ea5d640 -1 osd.3 153 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:09.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: st cls: Running main() from gmock_main.cc 2024-07-27T14:45:09.003 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-07-27T14:45:09.003 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [----------] Global test environment set-up. 2024-07-27T14:45:09.003 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [ OK ] NeoRadosCls.DNE (55947 ms) 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (106019 ms) 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [----------] 2 tests from NeoRadosCls (161966 ms total) 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [----------] Global test environment tear-down 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [==========] 2 tests from 1 test suite ran. (161966 ms total) 2024-07-27T14:45:09.004 INFO:tasks.workunit.client.0.smithi087.stdout: cls: [ PASSED ] 2 tests. 2024-07-27T14:45:10.141 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:10.050+0000 ea5d640 -1 osd.3 154 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:11.494 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:11.405+0000 ea5d640 -1 osd.3 155 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:12.827 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:12.814+0000 ea5d640 -1 osd.3 156 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:14.310 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:17.176 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.442 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:17.308 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.463 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:17.329 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.523 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:17.390 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.529 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:17.381 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.539 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:17.391 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.584 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:45:14.570+0000 ea5d640 -1 osd.3 157 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4522.0:6 24.d 24:b3c717c4:::4.obj:head [write 0~185 in=185b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72) 2024-07-27T14:45:14.596 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:17.463 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.681 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:17.533 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:14.684 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:17.551 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.658 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:42.524 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.684 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:42.551 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.722 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:42.574 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.730 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:42.582 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.768 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:42.634 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.778 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:42.644 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.877 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:42.743 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.892 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:05:42.744 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:39.930 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:42.796 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:54.060 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:54.048+0000 12a65640 -1 osd.6 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:55.312 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:55.301+0000 12a65640 -1 osd.6 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:56.421 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:56.414+0000 12a65640 -1 osd.6 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:57.437 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:57.433+0000 12a65640 -1 osd.6 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:58.468 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:58.461+0000 12a65640 -1 osd.6 198 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:59.506 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:02.373 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.577 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:02.444 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.633 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:45:59.613+0000 12a65640 -1 osd.6 199 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:45:59.636 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:02.488 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.637 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:02.490 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.646 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:02.512 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.676 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:02.542 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.746 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:02.598 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.751 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:02.617 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:45:59.830 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:02.696 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:01.123 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:46:01.036+0000 12a65640 -1 osd.6 200 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:46:02.226 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:46:02.148+0000 12a65640 -1 osd.6 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:46:03.182 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T14:46:03.170+0000 12a65640 -1 osd.6 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4452.0:21 15.16 15:6df9c5af:::19.obj:head [write 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout:arted 0x55f9e5105af0 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105640 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31e10 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f31f40 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f32050 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edae80 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105750 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb330 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51058a0 2024-07-27T14:46:12.163 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51059d0 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106a70 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb440 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb550 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb680 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51065c0 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106710 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106820 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106950 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106110 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106240 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51063b0 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105c60 2024-07-27T14:46:12.164 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105d90 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105ea0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5105fd0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107540 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107b40 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107690 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edaf90 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb0a0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edb1f0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51085f0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108c10 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108760 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51077a0 2024-07-27T14:46:12.165 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51078b0 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107a00 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106be0 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107090 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51071a0 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51072f0 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106cf0 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5106e60 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108d60 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107c90 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109320 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107420 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109430 2024-07-27T14:46:12.166 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107da0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108360 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51084d0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107eb0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108870 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51089c0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108ad0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109ea0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108e70 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5107fc0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51080f0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108240 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a4c0 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a010 2024-07-27T14:46:12.167 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5108f80 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109090 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51091e0 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510aac0 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510b0e0 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510b1f0 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510b320 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510ac30 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a120 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a270 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a380 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109540 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f284c0 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109650 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28630 2024-07-27T14:46:12.168 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28b00 2024-07-27T14:46:12.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109c70 2024-07-27T14:46:12.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109d80 2024-07-27T14:46:12.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28760 2024-07-27T14:46:12.169 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28870 2024-07-27T14:46:16.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:46:16.826+0000 ea5d640 -1 osd.0 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4742.0:2 100.17 100:e889c135:test-rados-api-smithi087-41444-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171) 2024-07-27T14:46:18.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T14:46:18.575+0000 ea5d640 -1 osd.0 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4742.0:2 100.17 100:e889c135:test-rados-api-smithi087-41444-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171) 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc api_cmd_pp: Running main() from gmock_main.cc 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (930 ms) 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-07-27T14:46:19.880 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (72 ms) 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (232027 ms) 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (233029 ms total) 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (233029 ms total) 2024-07-27T14:46:19.881 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-07-27T14:46:20.060 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:22.926 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.066 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:22.918 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.068 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:22.920 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.069 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:22.935 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.071 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:22.938 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.128 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:22.995 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.234 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:23.100 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.247 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:23.100 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:20.252 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:23.119 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:23.505 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:23.498+0000 ea5d640 -1 osd.3 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:24.584 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:24.566+0000 ea5d640 -1 osd.3 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:25.821 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:25.761+0000 ea5d640 -1 osd.3 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:27.066 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:27.017+0000 ea5d640 -1 osd.3 222 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:28.287 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:28.280+0000 ea5d640 -1 osd.3 223 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:29.871 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:29.817+0000 ea5d640 -1 osd.3 224 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:31.180 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:31.139+0000 ea5d640 -1 osd.3 225 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:32.199 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:32.169+0000 ea5d640 -1 osd.3 226 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4467.0:55 33.1e 33:7f03c1b6:test-rados-api-smithi087-41485-8::foo:head [watch ping cookie 94092260345744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188) 2024-07-27T14:46:33.562 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:33.377+0000 ea5d640 -1 osd.3 227 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 103.6 103.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e180) 2024-07-27T14:46:35.096 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:35.011+0000 ea5d640 -1 osd.3 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 103.6 103.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e180) 2024-07-27T14:46:36.539 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:36.474+0000 ea5d640 -1 osd.3 229 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 103.6 103.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e180) 2024-07-27T14:46:37.874 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:37.766+0000 ea5d640 -1 osd.3 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 103.6 103.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e180) 2024-07-27T14:46:40.212 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:43.079 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.283 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:43.149 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.349 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:43.215 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.355 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:43.221 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.359 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:43.212 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.368 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:43.220 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.396 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:43.263 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.484 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:43.350 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:40.485 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:06:43.337 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:46:41.704 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: Running main() from gmock_main.cc 2024-07-27T14:46:41.704 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] Global test environment set-up. 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (89857 ms) 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockShared (9553 ms) 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-07-27T14:46:41.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (5675 ms) 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (20481 ms) 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (9812 ms) 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.Unlock (9614 ms) 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (8630 ms) 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-07-27T14:46:41.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4678 ms) 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLock (158300 ms total) 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (24654 ms) 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (11490 ms) 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (6612 ms) 2024-07-27T14:46:41.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (22590 ms) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3694 ms) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (5634 ms) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (7602 ms) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (1903 ms) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (84179 ms total) 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: 2024-07-27T14:46:41.708 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] Global test environment tear-down 2024-07-27T14:46:41.709 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (255046 ms total) 2024-07-27T14:46:41.709 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ PASSED ] 16 tests. 2024-07-27T14:46:41.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:41.697+0000 12a65640 -1 osd.2 235 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:41.850 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T14:46:41.827+0000 ea5d640 -1 osd.3 233 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 103.6 103:602f83fe:::foo:head [write 0~128 in=128b,set-alloc-hint object_size 0 write_size 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180) 2024-07-27T14:46:42.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T14:46:42.777+0000 ea5d640 -1 osd.1 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4918.0:7 89.1s0 89:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207) 2024-07-27T14:46:43.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:43.234+0000 12a65640 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-07-27T14:46:43.713 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (54851 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (3109 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (58335 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (25706 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7544 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (15616 ms) 2024-07-27T14:46:43.714 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (18278 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7390 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (8587 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (3102 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (8334 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (3192 ms) 2024-07-27T14:46:43.715 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (17453 ms) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9979 ms) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (6181 ms) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (2942 ms) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (250599 ms total) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (256766 ms total) 2024-07-27T14:46:43.716 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-07-27T14:46:44.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:44.306+0000 12a65640 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:45.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:45.449+0000 12a65640 -1 osd.2 238 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:46.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:46.603+0000 12a65640 -1 osd.2 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:47.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:47.759+0000 12a65640 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:49.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:49.985+0000 12a65640 -1 osd.2 243 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:46:51.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T14:46:51.008+0000 12a65640 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4772.0:3 110.0 110:0384ab8e:::oid:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-07-27T14:47:00.370 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:03.236 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.449 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:03.315 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.512 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:03.364 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.515 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:03.367 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.535 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:03.401 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.543 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:03.410 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.568 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:03.434 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.618 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:03.470 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:00.660 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:03.526 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:16.673 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-07-27T14:47:16.673 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-07-27T14:47:16.673 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: seed 41451 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (36174 ms) 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (77497 ms) 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (11 ms) 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (33756 ms) 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-07-27T14:47:16.674 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (69688 ms) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (217126 ms total) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (8268 ms) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (288 ms) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2000 ms) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (49150 ms) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (59706 ms total) 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: 2024-07-27T14:47:16.678 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-07-27T14:47:16.679 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (289855 ms total) 2024-07-27T14:47:16.679 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: Running main() from gmock_main.cc 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] Global test environment set-up. 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-07-27T14:47:16.705 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.Stat (30166 ms) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (545 ms) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.StatNS (127844 ms) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (44 ms) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6183 ms) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 5 tests from LibRadosStat (164782 ms total) 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-07-27T14:47:16.706 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (82033 ms) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (27773 ms) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (15 ms) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1182 ms) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (111003 ms total) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] Global test environment tear-down 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (289913 ms total) 2024-07-27T14:47:16.707 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ PASSED ] 9 tests. 2024-07-27T14:47:20.450 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:23.316 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.453 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:23.320 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.511 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:23.363 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.514 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:23.366 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.525 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:23.391 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.535 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:23.402 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.561 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:23.427 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.602 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:23.454 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:20.634 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:23.500 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.802 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:27.668 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.879 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:27.746 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.894 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:27.746 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.896 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:27.748 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.929 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:27.781 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.933 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:27.800 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.959 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:27.825 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.964 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:27.830 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:24.999 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:27.865 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:34.176 INFO:tasks.workunit.client.0.smithi087.stdout:: started 0x55f9e4f289c0 2024-07-27T14:47:34.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51097c0 2024-07-27T14:47:34.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51098f0 2024-07-27T14:47:34.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5109a40 2024-07-27T14:47:34.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a610 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a740 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a870 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510a9a0 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28c30 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f291f0 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28d40 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28e90 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f28fc0 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29340 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29810 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29940 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510ad40 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510ae70 2024-07-27T14:47:34.177 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e510afa0 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29a90 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29f60 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a090 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29470 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f295c0 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a650 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a1a0 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a2d0 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a400 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a530 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ac30 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a780 2024-07-27T14:47:34.178 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a8b0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2a9c0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ab10 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b210 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ad60 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ae90 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2afc0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b340 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b880 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b9b0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29bc0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29cf0 2024-07-27T14:47:34.179 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f29e20 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2bae0 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2bfe0 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c130 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c730 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2bbf0 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2bd20 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2be50 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c880 2024-07-27T14:47:34.180 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c280 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c3d0 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c500 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c610 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2ce60 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2c9b0 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2cb20 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2cc50 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2cf70 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d490 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d610 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b490 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b5c0 2024-07-27T14:47:34.181 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2b6f0 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2e0a0 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2e1b0 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2e300 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2dbf0 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d740 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d890 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d9c0 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2dd00 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2de10 2024-07-27T14:47:34.182 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2df40 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc260 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc840 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc390 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc990 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc4a0 2024-07-27T14:47:34.183 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edc5f0 2024-07-27T14:47:36.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_ cmd: Running main() from gmock_main.cc 2024-07-27T14:47:36.295 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-07-27T14:47:36.295 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [----------] Global test environment set-up. 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6241 ms) 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (10569 ms) 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (292431 ms) 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [----------] 3 tests from NeoRadosCmd (309241 ms total) 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [----------] Global test environment tear-down 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [==========] 3 tests from 1 test suite ran. (309241 ms total) 2024-07-27T14:47:36.296 INFO:tasks.workunit.client.0.smithi087.stdout: cmd: [ PASSED ] 3 tests. 2024-07-27T14:47:40.598 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:43.450 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.600 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:43.452 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.608 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:43.475 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.616 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:43.482 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.666 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:07:43.518 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.675 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:43.541 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.678 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:43.544 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.714 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:43.580 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:47:40.774 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:43.640 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-07-27T14:48:00.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify_test_cb 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (16671 ms) 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94092260261952 err -107 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (60945 ms) 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94092260261952 err -107 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (44200 ms) 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 554050781184 cookie 94092261114192 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6838 ms) 2024-07-27T14:48:00.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 579820584961 cookie 94092261114192 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7941 ms) 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 639950127104 cookie 94092260345744 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (30079 ms) 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 970662608897 cookie 94092260345744 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 970662608897 cookie 94092260361680 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (103309 ms) 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 1108101562372 cookie 94092260361680 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 1112396529669 cookie 94092260361680 2024-07-27T14:48:00.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (7493 ms) 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94092260362880 err -107 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 4467 notify_id 1176821039105 cookie 94092260362880 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (23502 ms) 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to watch_notify2_test_errcb30 for disconnect notification ... cookie 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: 94092260362880 err -107 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (9492 ms) 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (310470 ms total) 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-07-27T14:48:00.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify_test_cb 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (10633 ms) 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (10633 ms total) 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (333470 ms total) 2024-07-27T14:48:00.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-07-27T14:48:00.939 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:03.805 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.002 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:03.869 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.066 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:03.933 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.073 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:03.925 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.076 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:03.928 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.078 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:03.945 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.126 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:03.993 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.196 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:04.048 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:01.205 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:04.072 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: Running main() from gmock_main.cc 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] Global test environment set-up. 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (48224 ms) 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-07-27T14:48:03.119 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (34028 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (16214 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (8005 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (106471 ms total) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (40022 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (31617 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (27108 ms) 2024-07-27T14:48:03.120 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (98747 ms total) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (8856 ms) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (38881 ms) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (15948 ms) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4805 ms) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (68490 ms total) 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-07-27T14:48:03.121 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (21675 ms) 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (17149 ms) 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (38824 ms total) 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] Global test environment tear-down 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (336368 ms total) 2024-07-27T14:48:03.122 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (146278 ms) 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (22172 ms) 2024-07-27T14:48:03.487 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (168450 ms total) 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (58472 ms) 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (9685 ms) 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5603 ms) 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5784 ms) 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-07-27T14:48:03.488 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746592997952 notify_id 1047972020227 notifier_gid 4742 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (6281 ms) 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1060856922113 notifier_gid 4742 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (2907 ms) 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1129576398850 notifier_gid 4742 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (19535 ms) 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1168231104515 notifier_gid 4742 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (8243 ms) 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1202590842884 notifier_gid 4742 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (11133 ms) 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-07-27T14:48:03.489 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1262720385029 notifier_gid 4742 2024-07-27T14:48:03.490 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (19104 ms) 2024-07-27T14:48:03.490 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-07-27T14:48:03.490 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: trying... 2024-07-27T14:48:03.490 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1322849927175 notifier_gid 4742 2024-07-27T14:48:03.490 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: timed out 2024-07-27T14:48:21.327 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:24.192 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.332 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:24.199 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.396 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:24.262 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.399 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:24.266 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.400 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:24.251 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.403 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:24.255 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.447 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:24.313 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.497 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:24.364 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:21.502 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:24.355 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:25.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_watc api_lock_pp: Running main() from gmock_main.cc 2024-07-27T14:48:25.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-07-27T14:48:25.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-07-27T14:48:25.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-07-27T14:48:25.769 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: seed 41079 2024-07-27T14:48:25.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-07-27T14:48:25.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (70196 ms) 2024-07-27T14:48:25.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-07-27T14:48:25.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (24548 ms) 2024-07-27T14:48:25.770 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (21117 ms) 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4086 ms) 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (32372 ms) 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-07-27T14:48:25.771 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (9393 ms) 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (44488 ms) 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (15741 ms) 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (221942 ms total) 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-07-27T14:48:25.772 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (9120 ms) 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4078 ms) 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (35585 ms) 2024-07-27T14:48:25.773 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (8777 ms) 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (18694 ms) 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (37219 ms) 2024-07-27T14:48:25.774 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (2978 ms) 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (7731 ms) 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (124182 ms total) 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (359096 ms total) 2024-07-27T14:48:25.775 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-07-27T14:48:36.379 INFO:tasks.workunit.client.0.smithi087.stdout:h_notify_pp: flushing 2024-07-27T14:48:36.379 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushed 2024-07-27T14:48:36.379 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (11175 ms) 2024-07-27T14:48:36.380 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-07-27T14:48:36.380 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: trying... 2024-07-27T14:48:36.380 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1361504632840 notifier_gid 4742 2024-07-27T14:48:36.380 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: timed out 2024-07-27T14:48:36.380 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushed 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (8517 ms) 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: List watches 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 2024-07-27T14:48:36.381 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1391569403913 notifier_gid 4742 2024-07-27T14:48:36.382 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 done 2024-07-27T14:48:36.382 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: watch_check 2024-07-27T14:48:36.382 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: unwatch2 2024-07-27T14:48:36.382 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-07-27T14:48:36.382 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: done 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (8044 ms) 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: List watches 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 94746593182800 notify_id 1417339207690 notifier_gid 4742 2024-07-27T14:48:36.383 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 done 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: watch_check 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: unwatch2 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: done 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (11510 ms) 2024-07-27T14:48:36.384 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (185993 ms total) 2024-07-27T14:48:36.385 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: 2024-07-27T14:48:36.385 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-07-27T14:48:36.385 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (369559 ms total) 2024-07-27T14:48:36.385 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-07-27T14:48:40.319 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: Running main() from gmock_main.cc 2024-07-27T14:48:40.319 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-07-27T14:48:40.319 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] Global test environment set-up. 2024-07-27T14:48:40.319 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-07-27T14:48:40.319 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: seed 41031 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (38892 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (46556 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (31208 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (10669 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2979 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (15196 ms) 2024-07-27T14:48:40.320 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (10029 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (18530 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (11374 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6637 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8141 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (7796 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3029 ms) 2024-07-27T14:48:40.321 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (67869 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (12414 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4174 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3643 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (774 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4930 ms) 2024-07-27T14:48:40.322 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (304840 ms total) 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (16282 ms) 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (37982 ms) 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2638 ms) 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (2320 ms) 2024-07-27T14:48:40.323 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-07-27T14:48:41.374 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:44.240 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.378 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:44.245 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.428 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:44.294 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.431 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:44.298 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.436 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:44.288 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.439 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:44.291 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.466 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:44.333 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.521 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:44.387 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:41.539 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:08:44.391 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:48:52.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] api_snapshots_pp: Running main() from gmock_main.cc 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: seed 41261 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (95936 ms) 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (12215 ms) 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (46966 ms) 2024-07-27T14:48:52.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (5533 ms) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (36611 ms) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (197261 ms total) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (14396 ms) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (18908 ms) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (64665 ms) 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-07-27T14:48:52.449 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (8166 ms) 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (22796 ms) 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3573-g57fd7103/rpm/el9/BUILD/ceph-19.0.0-3573-g57fd7103/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi087-41261-7 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: waiting for snaps to purge 2024-07-27T14:48:52.450 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (22373 ms) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (151304 ms total) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5312 ms) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5320 ms total) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (11185 ms) 2024-07-27T14:48:52.451 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] Li snapshots: Running main() from gmock_main.cc 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] Global test environment set-up. 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (70835 ms) 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (16524 ms) 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (70761 ms) 2024-07-27T14:48:54.696 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (17185 ms) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (29837 ms) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (205142 ms total) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (15662 ms) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (64014 ms) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (36992 ms) 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-07-27T14:48:54.697 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (17429 ms) 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (9074 ms) 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi087-42169-11. 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: Waiting for snaps to purge. 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (39225 ms) 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (182396 ms total) 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [----------] Global test environment tear-down 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (387538 ms total) 2024-07-27T14:48:54.698 INFO:tasks.workunit.client.0.smithi087.stdout: snapshots: [ PASSED ] 11 tests. 2024-07-27T14:48:56.189 INFO:tasks.workunit.client.0.smithi087.stdout:misc: started 0x55f9e4edc720 2024-07-27T14:48:56.189 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd440 2024-07-27T14:48:56.189 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edbdb0 2024-07-27T14:48:56.189 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d0a0 2024-07-27T14:48:56.189 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d1d0 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4f2d300 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edcae0 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd5b0 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edcbf0 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd1b0 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd300 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd700 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edcd00 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edce50 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edcf60 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd090 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd830 2024-07-27T14:48:56.190 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edde10 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edd960 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edbec0 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edbff0 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eddf60 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede520 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede070 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede1a0 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede2f0 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede650 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edec10 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede760 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edda70 2024-07-27T14:48:56.191 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eddb80 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eddcd0 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eded40 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf300 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edee50 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edef80 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf0d0 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf8e0 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf430 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf580 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edf6d0 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edfa30 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edff00 2024-07-27T14:48:56.192 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0030 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede870 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ede9a0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edeaf0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0610 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0160 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edfb60 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edfcb0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4edfdc0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0bf0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0740 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0890 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee09c0 2024-07-27T14:48:56.193 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0d00 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee11d0 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1320 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0270 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0380 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee04b0 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1450 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1920 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1a50 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0e30 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee0f60 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee10b0 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1b80 2024-07-27T14:48:56.194 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2050 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2180 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1580 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee16d0 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1800 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee22b0 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee27c0 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2910 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1cb0 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1de0 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee1f10 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2a40 2024-07-27T14:48:56.195 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2f10 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: Running main() from gmock_main.cc 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [----------] Global test environment set-up. 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (59480 ms) 2024-07-27T14:48:57.465 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (73269 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (32348 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (70020 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (10358 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (5730 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (29676 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (7279 ms) 2024-07-27T14:48:57.466 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (8206 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (17531 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (27455 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (14019 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (11459 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (23500 ms) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (390330 ms total) 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: 2024-07-27T14:48:57.467 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [----------] Global test environment tear-down 2024-07-27T14:48:57.468 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (390330 ms total) 2024-07-27T14:48:57.468 INFO:tasks.workunit.client.0.smithi087.stdout: read_operations: [ PASSED ] 14 tests. 2024-07-27T14:49:01.578 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:04.444 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.586 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:04.452 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.586 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:04.438 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.589 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:04.441 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.616 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:04.482 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.657 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:04.523 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.662 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:04.514 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.683 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:04.546 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:01.770 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:04.637 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:03.340 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: Running main() from gmock_main.cc 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] Global test environment set-up. 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (898 ms) 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (83 ms) 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (387881 ms) 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:53.472947+0000 mon.b [INF] from='client.? 172.21.15.87:0/2427531525' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-40983-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-07-27T14:49:03.341 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:53.525477+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-40983-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:54.515203+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-40983-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:55.771645+0000 mon.b [INF] from='client.? 172.21.15.87:0/454439962' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:55.920036+0000 client.admin [INF] onexx 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:55.926045+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:56.230993+0000 mon.a [WRN] Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:57.847269+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:59.921729+0000 mon.b [INF] from='client.? 172.21.15.87:0/454439962' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:59.926055+0000 client.admin [INF] threexx 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-07-27T14:48:59.929651+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7628 ms) 2024-07-27T14:49:03.342 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (396490 ms total) 2024-07-27T14:49:03.343 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: 2024-07-27T14:49:03.343 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] Global test environment tear-down 2024-07-27T14:49:03.343 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (396490 ms total) 2024-07-27T14:49:03.343 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ PASSED ] 4 tests. 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: Running main() from gmock_main.cc 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] Global test environment set-up. 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-07-27T14:49:06.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (64431 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (58401 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (98064 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.Checksum (3838 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2418 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2686 ms) 2024-07-27T14:49:06.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4018 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (4102 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.TruncTest (1973 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (28465 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3675 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RmXattr (32168 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4008 ms) 2024-07-27T14:49:06.311 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 14 tests from LibRadosIo (308247 ms total) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (43499 ms) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (6280 ms) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4984 ms) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (2405 ms) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (6418 ms) 2024-07-27T14:49:06.312 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (175 ms) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (1419 ms) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3756 ms) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (7568 ms) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (1683 ms) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 10 tests from LibRadosIoEC (78187 ms total) 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] Global test environment tear-down 2024-07-27T14:49:06.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [==========] 24 tests from 2 test suites ran. (399680 ms total) 2024-07-27T14:49:06.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ PASSED ] 24 tests. 2024-07-27T14:49:09.875 INFO:tasks.workunit.client.0.smithi087.stdout: io: Running main() from gmock_main.cc 2024-07-27T14:49:09.875 INFO:tasks.workunit.client.0.smithi087.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-07-27T14:49:09.875 INFO:tasks.workunit.client.0.smithi087.stdout: io: [----------] Global test environment set-up. 2024-07-27T14:49:09.875 INFO:tasks.workunit.client.0.smithi087.stdout: io: [----------] 14 tests from NeoRadosIo 2024-07-27T14:49:09.875 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.Limits (100288 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (70982 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.ReadOp (42289 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.SparseRead (66327 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.RoundTrip (11991 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (8942 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (8395 ms) 2024-07-27T14:49:09.876 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (26894 ms) 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (14640 ms) 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.Trunc (10768 ms) 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.Remove (6516 ms) 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (7939 ms) 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-07-27T14:49:09.877 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.RmXattr (5494 ms) 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ OK ] NeoRadosIo.GetXattrs (21327 ms) 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [----------] 14 tests from NeoRadosIo (402793 ms total) 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [----------] Global test environment tear-down 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [==========] 14 tests from 1 test suite ran. (402793 ms total) 2024-07-27T14:49:09.878 INFO:tasks.workunit.client.0.smithi087.stdout: io: [ PASSED ] 14 tests. 2024-07-27T14:49:16.041 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: Running main() from gmock_main.cc 2024-07-27T14:49:16.041 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-07-27T14:49:16.041 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [----------] Global test environment set-up. 2024-07-27T14:49:16.041 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-07-27T14:49:16.041 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (408969 ms) 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [----------] 1 test from neocls_handler_error (408969 ms total) 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [----------] Global test environment tear-down 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [==========] 1 test from 1 test suite ran. (408969 ms total) 2024-07-27T14:49:16.042 INFO:tasks.workunit.client.0.smithi087.stdout: handler_error: [ PASSED ] 1 test. 2024-07-27T14:49:21.907 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:24.773 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.916 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:24.783 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.969 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:24.835 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.971 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:24.837 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.985 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:24.837 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.987 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:24.840 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:21.999 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:24.865 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:22.055 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:24.921 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:22.063 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:24.915 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.777 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:26.629 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.778 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:26.630 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.795 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:26.647 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.879 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:26.745 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.880 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:26.746 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:23.902 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:26.769 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:24.420 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:27.287 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:24.421 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:27.288 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:24.452 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:27.319 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: LibRadosIoECPP.RoundTripPP2 (1600 ms) 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4381 ms) 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (2616 ms) 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3025 ms) 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (2827 ms) 2024-07-27T14:49:28.613 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (1513 ms) 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2104 ms) 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3972 ms) 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9566 ms) 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-07-27T14:49:28.614 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3961 ms) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8821 ms) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1531 ms) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (1197 ms) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (106336 ms total) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] Global test environment tear-down 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (421972 ms total) 2024-07-27T14:49:28.615 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-07-27T14:49:29.885 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: starting. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: creating pool ceph_test_rados_open_pools_parallel.smithi087-41748 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: created object 0... 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: created object 25... 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: created object 49... 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: finishing. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[41855]: shutting down. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[41860]: starting. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[41860]: rados_pool_create. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[41860]: rados_ioctx_create. 2024-07-27T14:49:29.886 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[41860]: shutting down. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: starting. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: creating pool ceph_test_rados_open_pools_parallel.smithi087-41748 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: created object 0... 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: created object 25... 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: created object 49... 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: finishing. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[44232]: shutting down. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[44233]: starting. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[44233]: rados_pool_create. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[44233]: rados_ioctx_create. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[44233]: shutting down. 2024-07-27T14:49:29.887 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-07-27T14:49:29.888 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-07-27T14:49:29.888 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-07-27T14:49:40.083 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: starting. 2024-07-27T14:49:40.083 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: creating pool ceph_test_rados_delete_pools_parallel.smithi087-41815 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: created object 0... 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: created object 25... 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: created object 49... 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: finishing. 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[41870]: shutting down. 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[41871]: starting. 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[41871]: deleting pool ceph_test_rados_delete_pools_parallel.smithi087-41815 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[41871]: shutting down. 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: starting. 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: creating pool ceph_test_rados_delete_pools_parallel.smithi087-41815 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: created object 0... 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: created object 25... 2024-07-27T14:49:40.084 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: created object 49... 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: finishing. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[44442]: shutting down. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: starting. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: listing objects. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: listed object 0... 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: listed object 25... 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: saw 50 objects 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[44444]: shutting down. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[44443]: starting. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[44443]: deleting pool ceph_test_rados_delete_pools_parallel.smithi087-41815 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[44443]: shutting down. 2024-07-27T14:49:40.085 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-07-27T14:49:40.086 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-07-27T14:49:40.086 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-07-27T14:49:41.822 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:44.688 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.826 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:44.692 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.895 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:44.747 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.902 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:44.754 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.904 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:44.771 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.907 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:44.773 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.961 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:44.827 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.977 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:09:44.829 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:41.988 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:44.855 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout:bRadosSnapshotsECPP.SnapRemovePP (8318 ms) 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (7778 ms) 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (8589 ms) 2024-07-27T14:49:42.316 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (35870 ms total) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (12629 ms) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5449 ms) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5847 ms) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (23925 ms total) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (435584 ms total) 2024-07-27T14:49:42.317 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-07-27T14:49:42.318 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-07-27T14:49:42.318 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-07-27T14:50:02.121 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:04.987 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.199 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:05.050 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.200 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:05.052 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.206 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:05.072 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.237 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:05.103 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.264 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:05.116 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.278 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:05.145 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.288 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:05.154 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:02.348 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:05.215 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:10.267 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-07-27T14:50:10.267 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] Global test environment set-up. 2024-07-27T14:50:10.267 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 7 tests from LibRadosList 2024-07-27T14:50:10.267 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjects (239577 ms) 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (60065 ms) 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo2,foo3 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo2 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo3 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo4,foo5 2024-07-27T14:50:10.268 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo4 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo5 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo6,foo7 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo7 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo6 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo4 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo5 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo7 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo6 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo1 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo1 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo2 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo3 2024-07-27T14:50:10.269 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (124320 ms) 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 1 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 10 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 13 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 7 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 14 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 0 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 15 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 11 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 5 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 8 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 6 0 2024-07-27T14:50:10.270 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 3 0 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 4 0 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 12 0 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 9 0 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2 0 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (17050 ms) 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: x cursor=MIN 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-07-27T14:50:10.271 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: FIRST> seek to MIN oid=1 2024-07-27T14:50:10.272 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-07-27T14:50:10.273 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:566253c9:::13:head 2024-07-27T14:50:10.274 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-07-27T14:50:10.274 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-07-27T14:50:10.274 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-07-27T14:50:10.274 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=14 cursor=8:62a1935d: api_aio: Running main() from gmock_main.cc 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] Global test environment set-up. 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5135 ms) 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (78582 ms) 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (26142 ms) 2024-07-27T14:50:16.440 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (50208 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (28018 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (80367 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (26366 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (6496 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (16729 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (30806 ms) 2024-07-27T14:50:16.441 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (36920 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (15241 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (13983 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4838 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.Flush (3418 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4205 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4806 ms) 2024-07-27T14:50:16.442 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (6446 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3624 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4540 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3358 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3371 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3425 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4572 ms) 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-07-27T14:50:16.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4831 ms) 2024-07-27T14:50:16.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-07-27T14:50:16.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3391 ms) 2024-07-27T14:50:16.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 26 tests from LibRadosAio (469818 ms total) 2024-07-27T14:50:16.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: 2024-07-27T14:50:16.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.Sapi_misc: started 0x55f9e4ee3060 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3640 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2b70 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2c80 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2dd0 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3790 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3190 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2400 2024-07-27T14:50:18.203 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2550 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee2680 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3d70 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee38c0 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee39f0 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3b00 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3c50 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4350 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3ea0 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3ff0 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4100 2024-07-27T14:50:18.204 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4480 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4a40 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4590 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee32a0 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee33d0 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee3520 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5040 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4b90 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee46a0 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee47d0 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4900 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5620 2024-07-27T14:50:18.205 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5170 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee52c0 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee53f0 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5750 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5c20 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5d30 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4ca0 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4dd0 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee4f00 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5e60 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6330 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6440 2024-07-27T14:50:18.206 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5880 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee59d0 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6a00 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6550 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6680 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6790 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee68a0 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6fe0 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6b30 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6c60 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6d90 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee6ec0 2024-07-27T14:50:18.207 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee75c0 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7110 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7260 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7390 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee76f0 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7bc0 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7d10 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee5f90 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee60c0 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee61f0 2024-07-27T14:50:18.208 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7e20 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee82d0 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8420 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7820 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7950 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7a80 2024-07-27T14:50:18.209 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8570 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8a20 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8b70 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee7f30 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8060 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8190 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8ca0 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9170 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee92c0 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8680 2024-07-27T14:50:18.210 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee87b0 2024-07-27T14:50:22.017 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:24.883 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.019 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:24.886 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.072 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:24.939 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.074 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:24.940 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.077 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:24.929 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.080 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:24.930 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.118 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:24.984 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.130 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:24.982 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:22.143 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:25.009 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:30.670 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: starting. 2024-07-27T14:50:30.670 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: creating pool ceph_test_rados_list_parallel.smithi087-41745 2024-07-27T14:50:30.670 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: created object 0... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: created object 25... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: created object 49... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: finishing. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[41820]: shutting down. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: starting. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: listing objects. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: listed object 0... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: listed object 25... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: saw 50 objects 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[41821]: shutting down. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: starting. 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: creating pool ceph_test_rados_list_parallel.smithi087-41745 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: created object 0... 2024-07-27T14:50:30.671 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: created object 25... 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: created object 49... 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: finishing. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[44305]: shutting down. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: starting. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: listing objects. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: listed object 0... 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: listed object 25... 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: saw 45 objects 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[44306]: shutting down. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: starting. 2024-07-27T14:50:30.672 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: removed 25 objects... 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: removed half of the objects 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: removed 50 objects... 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: removed 50 objects 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[44307]: shutting down. 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: starting. 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: creating pool ceph_test_rados_list_parallel.smithi087-41745 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: created object 0... 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: created object 25... 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: created object 49... 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: finishing. 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[44793]: shutting down. 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.673 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: starting. 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: listing objects. 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: listed object 0... 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: listed object 25... 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: listed object 50... 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: saw 55 objects 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[44794]: shutting down. 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.674 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: starting. 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: added 25 objects... 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: added half of the objects 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: added 50 objects... 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: added 50 objects 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[44795]: shutting down. 2024-07-27T14:50:30.675 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:30.676 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:35.394 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: **********************::14:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:863748b0:::15:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-07-27T14:50:35.395 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-07-27T14:50:35.396 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:e960b815:::9:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:566253c9:::13:head 2024-07-27T14:50:35.397 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=13 expected=13 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=10 expected=10 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=14 expected=14 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-07-27T14:50:35.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=0 expected=0 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=5 expected=5 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=11 expected=11 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:863748b0:::15:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=15 expected=15 2024-07-27T14:50:35.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-07-27T14:50:35.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-07-27T14:50:35.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-07-27T14:50:35.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=1 expected=1 2024-07-27T14:50:42.022 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:44.888 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.133 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:44.999 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.138 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:44.990 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.139 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:45.005 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.140 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:45.007 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.145 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:44.997 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.193 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:45.059 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.193 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:45.059 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:42.202 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:10:45.054 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:de5d7c5f:::1********* 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: starting. 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: creating pool ceph_test_rados_list_parallel.smithi087-41745 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: created object 0... 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: created object 25... 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: created object 49... 2024-07-27T14:50:53.934 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: finishing. 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[44940]: shutting down. 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: starting. 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listing objects. 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listed object 0... 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listed object 25... 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listed object 50... 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listed object 75... 2024-07-27T14:50:53.935 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: listed object 100... 2024-07-27T14:50:53.937 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: saw 101 objects 2024-07-27T14:50:53.937 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[44941]: shutting down. 2024-07-27T14:50:53.937 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.937 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: starting. 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: added 25 objects... 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: added half of the objects 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: added 50 objects... 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: added 50 objects 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[44942]: shutting down. 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: starting. 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: added 25 objects... 2024-07-27T14:50:53.938 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: added half of the objects 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: added 50 objects... 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: added 50 objects 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[44943]: shutting down. 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: starting. 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: removed 25 objects... 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: removed half of the objects 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: removed 50 objects... 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: removed 50 objects 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[44944]: shutting down. 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.939 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: starting. 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: creating pool ceph_test_rados_list_parallel.smithi087-41745 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: created object 0... 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: created object 25... 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: created object 49... 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: finishing. 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[45092]: shutting down. 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.940 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:50:53.941 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: starting. 2024-07-27T14:50:53.941 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listing objects. 2024-07-27T14:51:00.352 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 0... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 25... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 50... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 75... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 100... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: listed object 125... 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: saw 150 objects 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[45093]: shutting down. 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.353 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: starting. 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: added 25 objects... 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: added half of the objects 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: added 50 objects... 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: added 50 objects 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[45094]: shutting down. 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-07-27T14:51:00.354 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******* SUCCESS ********** 2024-07-27T14:51:02.162 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:05.028 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.224 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:05.076 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.225 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:05.092 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.225 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:05.078 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.230 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:05.097 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.248 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:05.113 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.289 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:05.141 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.296 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:05.162 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:02.317 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:05.183 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.437 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:25.302 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.439 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:25.305 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.505 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:25.357 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.505 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:25.372 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.506 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:25.358 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.521 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:25.386 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.526 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:25.392 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.568 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:25.419 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:22.586 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:25.452 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.855 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:27.721 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.859 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:27.725 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.879 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:27.745 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.917 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:27.769 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.918 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:27.784 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.919 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:27.785 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.919 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:27.771 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.944 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:27.810 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:24.947 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:27.799 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.469 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:40.335 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.474 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:40.340 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.511 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:40.377 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.526 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:40.378 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.527 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:40.379 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.547 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:40.413 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.548 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:40.415 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.570 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:40.422 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:37.598 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:40.464 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.387 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:45.253 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.392 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:45.259 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.419 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:45.285 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.443 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:45.295 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.445 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:45.297 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.462 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:45.314 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.470 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:45.336 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.474 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:45.340 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:42.502 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:45.368 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:46.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee88e0 2024-07-27T14:51:46.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee93f0 2024-07-27T14:51:46.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9540 2024-07-27T14:51:46.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9b40 2024-07-27T14:51:46.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9690 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee97e0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee98f0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9a20 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea100 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9c50 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eeab90 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9da0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9ef0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea6e0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea810 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea940 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eeaa50 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51133d0 2024-07-27T14:51:46.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea230 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea360 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea490 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4eea5c0 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51139b0 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113500 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113650 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113780 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113ae0 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113c30 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114230 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113d80 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114360 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51144b0 2024-07-27T14:51:46.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8dd0 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee8f00 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e4ee9030 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51145c0 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114ab0 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114c00 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e51146d0 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114820 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114950 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114d30 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114e80 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115480 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5114fd0 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115100 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115230 2024-07-27T14:51:46.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115590 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115a60 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5115b70 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55f9e5113e90 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:46.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: Running main() from gmock_main.cc 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [----------] Global test environment set-up. 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [----------] 3 tests from NeoradosList 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ OK ] NeoradosList.ListObjects (192453 ms) 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ OK ] NeoradosList.ListObjectsNS (201934 ms) 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-07-27T14:51:52.076 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ OK ] NeoradosList.ListObjectsMany (170575 ms) 2024-07-27T14:51:52.077 INFO:tasks.workunit.client.0.smithi087.stdout: list: [----------] 3 tests from NeoradosList (564962 ms total) 2024-07-27T14:51:52.077 INFO:tasks.workunit.client.0.smithi087.stdout: list: 2024-07-27T14:51:52.077 INFO:tasks.workunit.client.0.smithi087.stdout: list: [----------] Global test environment tear-down 2024-07-27T14:51:52.077 INFO:tasks.workunit.client.0.smithi087.stdout: list: [==========] 3 tests from 1 test suite ran. (564964 ms total) 2024-07-27T14:51:52.077 INFO:tasks.workunit.client.0.smithi087.stdout: list: [ PASSED ] 3 tests. 2024-07-27T14:51:52.575 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:55.441 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.579 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:55.446 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.621 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:55.487 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.657 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:55.509 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.658 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:55.524 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.658 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:55.510 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.662 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:55.528 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.700 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:11:55.552 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:52.707 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:55.574 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:51:55.659 INFO:tasks.workunit.client.0.smithi087.stdout:2:head 2024-07-27T14:51:55.659 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=12 expected=12 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=4 expected=4 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=3 expected=3 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-07-27T14:51:55.660 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=6 expected=6 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=8 expected=8 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=2 expected=2 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:e960b815:::9:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-07-27T14:51:55.661 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=9 expected=9 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=7 expected=7 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (48900 ms) 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (11116 ms) 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-07-27T14:51:55.662 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (48692 ms) 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 7 tests from LibRadosList (549720 ms total) 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (3188 ms) 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo2,foo3 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo2 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo3 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo4,foo5 2024-07-27T14:51:55.663 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo4 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo5 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo6,foo7 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo7 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo6 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo4 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo5 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo7 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo6 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo1 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo1 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo2 2024-07-27T14:51:55.664 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo3 2024-07-27T14:51:55.665 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3408 ms) 2024-07-27T14:51:55.665 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-07-27T14:51:55.665 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 1 0 2024-07-27T14:51:55.665 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 10 0 2024-07-27T14:51:55.665 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 13 0 2024-07-27T14:52:02.475 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:05.341 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.480 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:05.346 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.482 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:05.333 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.488 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:05.340 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.522 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:05.387 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.522 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:05.374 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.542 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:05.408 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.547 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:05.413 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:02.578 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:05.444 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:06.091 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: Running main() from gmock_main.cc 2024-07-27T14:52:06.091 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [----------] Global test environment set-up. 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (121455 ms) 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (199321 ms) 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (258210 ms) 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [----------] 3 tests from NeoradosECList (578986 ms total) 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [----------] Global test environment tear-down 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (578986 ms total) 2024-07-27T14:52:06.092 INFO:tasks.workunit.client.0.smithi087.stdout: ec_list: [ PASSED ] 3 tests. 2024-07-27T14:52:06.173 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 7 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 14 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 0 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 15 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 11 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 5 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 8 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 6 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 3 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 4 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 12 0 2024-07-27T14:52:06.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 9 0 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2 0 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (6541 ms) 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 3 tests from LibRadosListEC (13137 ms total) 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (4427 ms) 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 1 test from LibRadosListNP (4427 ms total) 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] Global test environment tear-down 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [==========] 11 tests from 3 test suites ran. (579489 ms total) 2024-07-27T14:52:06.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ PASSED ] 11 tests. 2024-07-27T14:52:07.650 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:10.516 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.654 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:10.521 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.689 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:10.555 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.716 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:10.568 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.722 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:10.574 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.727 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:10.594 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.750 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:10.602 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.759 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:10.626 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:07.786 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:10.653 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 1 sec 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 512 aios 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5071740 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a642b0 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eeb5a0 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5994820 2024-07-27T14:52:08.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4d76660 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510f3e0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510fe30 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2e8c0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f330 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2fd50 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f0a0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30ca0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2fb60 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31bf0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f309a0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f327b0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110de0 2024-07-27T14:52:08.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31930 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111940 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51123f0 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110b10 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51132c0 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112180 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda260 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edad70 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113030 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edbca0 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f1b0 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31810 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30ab0 2024-07-27T14:52:08.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda100 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda9e0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ef40 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e50690d0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edaaf0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edac00 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110c20 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f440 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111f00 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f590 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f6a0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f7b0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2f8c0 2024-07-27T14:52:08.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110ef0 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111510 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51116c0 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112500 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112b60 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112c70 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112d80 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111a50 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51117d0 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111b80 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111cd0 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30db0 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31410 2024-07-27T14:52:08.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31520 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ea50 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ed9d60 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ed9eb0 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ed9fe0 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30f40 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31050 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31180 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31290 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111020 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2eb60 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ec70 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31a40 2024-07-27T14:52:08.222 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ed80 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e50703a0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e50704d0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e50705e0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31630 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e50706f0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda410 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111130 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111240 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5111390 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a57220 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a57390 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a574a0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a575b0 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e59b4130 2024-07-27T14:52:08.223 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e59b4280 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e59b43f0 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e59b4560 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112630 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112740 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5112850 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51129a0 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5068900 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5068a10 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5068b20 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5068c70 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110900 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a54810 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a54920 2024-07-27T14:52:08.224 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a54a30 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5a54b40 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2fe60 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30480 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30590 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f306a0 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f307f0 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ffd0 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda520 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda630 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eda780 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f321b0 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f322e0 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f32430 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f32540 2024-07-27T14:52:08.225 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f32650 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb7e0 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb930 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edba80 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e506e760 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e506e870 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e506e980 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e506ead0 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31d00 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f300e0 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f301f0 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f30340 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510ffa0 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110450 2024-07-27T14:52:08.226 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5110560 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51106b0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51100b0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51101c0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51102d0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51054f0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105af0 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105640 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31e10 2024-07-27T14:52:08.227 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f31f40 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f32050 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edae80 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105750 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb330 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51058a0 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51059d0 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106a70 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb440 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb550 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb680 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51065c0 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106710 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106820 2024-07-27T14:52:08.228 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106950 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106110 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106240 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51063b0 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105c60 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105d90 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105ea0 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5105fd0 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107540 2024-07-27T14:52:08.229 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107b40 2024-07-27T14:52:08.230 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107690 2024-07-27T14:52:08.230 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edaf90 2024-07-27T14:52:08.230 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb0a0 2024-07-27T14:52:08.230 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edb1f0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51085f0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108c10 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108760 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51077a0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51078b0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107a00 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106be0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107090 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51071a0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51072f0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106cf0 2024-07-27T14:52:08.231 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5106e60 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108d60 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107c90 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109320 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107420 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109430 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107da0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108360 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51084d0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107eb0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108870 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51089c0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108ad0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109ea0 2024-07-27T14:52:08.232 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108e70 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5107fc0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51080f0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108240 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a4c0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a010 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5108f80 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109090 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51091e0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510aac0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510b0e0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510b1f0 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510b320 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510ac30 2024-07-27T14:52:08.233 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a120 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a270 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a380 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109540 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f284c0 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109650 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28630 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28b00 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109c70 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109d80 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28760 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28870 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f289c0 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51097c0 2024-07-27T14:52:08.234 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51098f0 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5109a40 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a610 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a740 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a870 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510a9a0 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28c30 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f291f0 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28d40 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28e90 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f28fc0 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29340 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29810 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29940 2024-07-27T14:52:08.235 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510ad40 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510ae70 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e510afa0 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29a90 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29f60 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a090 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29470 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f295c0 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a650 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a1a0 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a2d0 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a400 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a530 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ac30 2024-07-27T14:52:08.236 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a780 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a8b0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2a9c0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ab10 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b210 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ad60 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ae90 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2afc0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b340 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b880 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b9b0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29bc0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29cf0 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f29e20 2024-07-27T14:52:08.237 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2bae0 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2bfe0 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c130 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c730 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2bbf0 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2bd20 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2be50 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c880 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c280 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c3d0 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c500 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c610 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2ce60 2024-07-27T14:52:08.238 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2c9b0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2cb20 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2cc50 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2cf70 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d490 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d610 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b490 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b5c0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2b6f0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2e0a0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2e1b0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2e300 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2dbf0 2024-07-27T14:52:08.239 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d740 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d890 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d9c0 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2dd00 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2de10 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2df40 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc260 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc840 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc390 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc990 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc4a0 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc5f0 2024-07-27T14:52:08.240 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edc720 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd440 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edbdb0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d0a0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d1d0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4f2d300 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edcae0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd5b0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edcbf0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd1b0 2024-07-27T14:52:08.241 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd300 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd700 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edcd00 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edce50 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edcf60 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd090 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd830 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edde10 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edd960 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edbec0 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edbff0 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eddf60 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede520 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede070 2024-07-27T14:52:08.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede1a0 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede2f0 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede650 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edec10 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede760 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edda70 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eddb80 2024-07-27T14:52:08.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eddcd0 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eded40 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf300 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edee50 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edef80 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf0d0 2024-07-27T14:52:08.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf8e0 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf430 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf580 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edf6d0 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edfa30 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edff00 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0030 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede870 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ede9a0 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edeaf0 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0610 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0160 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edfb60 2024-07-27T14:52:08.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edfcb0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4edfdc0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0bf0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0740 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0890 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee09c0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0d00 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee11d0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1320 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0270 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0380 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee04b0 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1450 2024-07-27T14:52:08.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1920 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1a50 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0e30 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee0f60 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee10b0 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1b80 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2050 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2180 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1580 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee16d0 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1800 2024-07-27T14:52:08.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee22b0 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee27c0 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2910 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1cb0 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1de0 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee1f10 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2a40 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2f10 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3060 2024-07-27T14:52:08.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3640 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2b70 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2c80 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2dd0 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3790 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3190 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2400 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2550 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee2680 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3d70 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee38c0 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee39f0 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3b00 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3c50 2024-07-27T14:52:08.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4350 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3ea0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3ff0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4100 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4480 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4a40 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4590 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee32a0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee33d0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee3520 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5040 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4b90 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee46a0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee47d0 2024-07-27T14:52:08.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4900 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5620 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5170 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee52c0 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee53f0 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5750 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5c20 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5d30 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4ca0 2024-07-27T14:52:08.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4dd0 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee4f00 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5e60 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6330 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6440 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5880 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee59d0 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6a00 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6550 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6680 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6790 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee68a0 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6fe0 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6b30 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6c60 2024-07-27T14:52:08.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6d90 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee6ec0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee75c0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7110 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7260 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7390 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee76f0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7bc0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7d10 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee5f90 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee60c0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee61f0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7e20 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee82d0 2024-07-27T14:52:08.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8420 2024-07-27T14:52:08.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7820 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7950 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7a80 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8570 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8a20 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8b70 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee7f30 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8060 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8190 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8ca0 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9170 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee92c0 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8680 2024-07-27T14:52:08.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee87b0 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee88e0 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee93f0 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9540 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9b40 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9690 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee97e0 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee98f0 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9a20 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea100 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9c50 2024-07-27T14:52:08.260 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eeab90 2024-07-27T14:52:24.874 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:27.740 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:24.884 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:27.751 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.055 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:27.921 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.603 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:28.469 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.608 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:28.472 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.689 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:28.540 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.702 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:28.554 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.767 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:28.633 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:25.850 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:28.702 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.513 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:32.379 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.522 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:32.386 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.710 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:32.575 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.819 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:32.671 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.825 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:32.676 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:29.871 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:12:32.723 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:30.353 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:33.219 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:30.413 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:33.276 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:30.486 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:33.353 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: wai api_c_write_operations: Running main() from gmock_main.cc 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-07-27T14:52:33.542 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (28613 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (14866 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (19818 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (16782 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (14710 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (29734 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (482112 ms) 2024-07-27T14:52:33.543 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (606636 ms total) 2024-07-27T14:52:33.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: 2024-07-27T14:52:33.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-07-27T14:52:33.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (606636 ms total) 2024-07-27T14:52:33.544 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-07-27T14:52:35.856 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: Running main() from gmock_main.cc 2024-07-27T14:52:35.856 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-07-27T14:52:35.856 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [----------] Global test environment set-up. 2024-07-27T14:52:35.856 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-07-27T14:52:35.856 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (32148 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (12545 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (17045 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (54780 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (49690 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (15363 ms) 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-07-27T14:52:35.857 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (427116 ms) 2024-07-27T14:52:35.858 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (608687 ms total) 2024-07-27T14:52:35.858 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: 2024-07-27T14:52:35.858 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [----------] Global test environment tear-down 2024-07-27T14:52:35.858 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (608687 ms total) 2024-07-27T14:52:35.858 INFO:tasks.workunit.client.0.smithi087.stdout: write_operations: [ PASSED ] 7 tests. 2024-07-27T14:52:54.520 INFO:tasks.workunit.client.0.smithi087.stdout:impleWrite 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8813 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8067 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8067 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6879 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8453 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8079 ms) 2024-07-27T14:52:54.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8544 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7622 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7359 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8788 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7861 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7692 ms) 2024-07-27T14:52:54.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (8024 ms) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (8249 ms) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7809 ms) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (37771 ms) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (158077 ms total) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] Global test environment tear-down 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (627895 ms total) 2024-07-27T14:52:54.523 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ PASSED ] 42 tests. 2024-07-27T14:53:00.288 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:03.154 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.298 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:03.164 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.368 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:03.235 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.792 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:03.644 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.796 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:03.648 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.845 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:03.697 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.859 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:03.725 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.873 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:03.739 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:00.944 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:03.810 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.059 INFO:tasks.workunit.client.0.smithi087.stdout:ting 0x55f9e4ee9da0 2024-07-27T14:53:18.059 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9ef0 2024-07-27T14:53:18.060 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea6e0 2024-07-27T14:53:18.060 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea810 2024-07-27T14:53:18.060 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea940 2024-07-27T14:53:18.060 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eeaa50 2024-07-27T14:53:18.060 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51133d0 2024-07-27T14:53:18.061 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea230 2024-07-27T14:53:18.061 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea360 2024-07-27T14:53:18.061 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea490 2024-07-27T14:53:18.061 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4eea5c0 2024-07-27T14:53:18.061 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51139b0 2024-07-27T14:53:18.062 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113500 2024-07-27T14:53:18.062 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113650 2024-07-27T14:53:18.063 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113780 2024-07-27T14:53:18.063 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113ae0 2024-07-27T14:53:18.063 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113c30 2024-07-27T14:53:18.063 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114230 2024-07-27T14:53:18.063 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113d80 2024-07-27T14:53:18.064 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114360 2024-07-27T14:53:18.064 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51144b0 2024-07-27T14:53:18.064 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8dd0 2024-07-27T14:53:18.064 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee8f00 2024-07-27T14:53:18.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e4ee9030 2024-07-27T14:53:18.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51145c0 2024-07-27T14:53:18.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114ab0 2024-07-27T14:53:18.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114c00 2024-07-27T14:53:18.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e51146d0 2024-07-27T14:53:18.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114820 2024-07-27T14:53:18.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114950 2024-07-27T14:53:18.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114d30 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114e80 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115480 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5114fd0 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115100 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115230 2024-07-27T14:53:18.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115590 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115a60 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5115b70 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55f9e5113e90 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: done. 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (577978 ms) 2024-07-27T14:53:18.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (577978 ms total) 2024-07-27T14:53:18.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:53:18.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-07-27T14:53:18.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-07-27T14:53:18.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-07-27T14:53:18.069 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-07-27T14:53:18.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.Exec (669 ms) 2024-07-27T14:53:18.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-07-27T14:53:18.070 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (112 ms) 2024-07-27T14:53:18.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-07-27T14:53:18.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (52 ms) 2024-07-27T14:53:18.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-07-27T14:53:18.071 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4941 ms) 2024-07-27T14:53:18.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-07-27T14:53:18.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-07-27T14:53:18.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-07-27T14:53:18.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-07-27T14:53:18.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (59153 ms) 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 8 tests from LibRadosMisc (64927 ms total) 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] Global test environment tear-down 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (651314 ms total) 2024-07-27T14:53:18.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ PASSED ] 12 tests. 2024-07-27T14:53:18.110 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:20.976 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.162 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:21.028 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.164 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:21.016 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.165 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:21.017 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.180 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:21.046 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.182 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:21.034 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.195 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:21.062 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.199 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:21.065 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:18.240 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:21.106 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:22.797 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-07-27T14:53:22.797 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: seed 41212 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (14 ms) 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (64860 ms) 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (96751 ms) 2024-07-27T14:53:22.798 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (24784 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (13678 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1837 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4885 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3546 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1208 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (8691 ms) 2024-07-27T14:53:22.799 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (1788 ms) 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4285 ms) 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (10069 ms) 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: osd_max_attr_size = 0 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (279033 ms) 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (26827 ms) 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-07-27T14:53:22.800 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: done waiting, doing copies 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: waiting for final deep scrubs... 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: done waiting 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (91974 ms) 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (113 ms) 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (58 ms) 2024-07-27T14:53:22.801 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (15699 ms) 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-07-27T14:53:22.802 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-07-27T14:53:22.803 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-07-27T14:53:27.987 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:30.853 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.062 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:30.929 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.096 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:30.962 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.122 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:30.974 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.127 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:30.979 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.136 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:31.002 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.145 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:31.011 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.162 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:31.014 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:28.182 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:31.048 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.249 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:41.114 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.309 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:41.161 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.310 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:41.176 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.310 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:41.163 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.322 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:41.189 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.326 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:41.192 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.352 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:41.204 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.370 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:41.236 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:38.391 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:41.257 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.128 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:50.994 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.161 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:51.028 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.177 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:51.029 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.178 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:51.030 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.191 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:51.058 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.196 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:51.062 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.203 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:51.069 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.209 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:51.061 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:48.227 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:51.093 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.256 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:56.122 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.314 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:56.180 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.330 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:13:56.197 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.358 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:56.210 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.359 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:56.226 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.361 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:56.228 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.363 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:56.215 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.383 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:13:56.249 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:53.383 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:13:56.235 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:53:54.398 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (650100 ms total) 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2137 ms) 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2137 ms total) 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (1214 ms) 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (691 ms) 2024-07-27T14:53:54.399 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (1905 ms total) 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (500 ms) 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (152 ms) 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (652 ms total) 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (605 ms) 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-07-27T14:53:54.400 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (64 ms) 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (669 ms total) 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1643 ms) 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1644 ms total) 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (687654 ms total) 2024-07-27T14:53:54.401 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-07-27T14:54:08.322 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:11.188 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.326 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:11.193 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.363 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:11.229 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.373 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:11.225 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.374 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:11.226 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.375 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:11.241 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.376 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:11.242 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.411 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:11.263 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:08.432 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:11.298 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.489 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:26.355 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.496 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:26.362 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.537 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:26.403 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.551 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:26.403 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.553 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:26.404 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.555 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:26.421 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.558 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:26.424 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.593 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:26.445 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:23.613 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:26.479 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.604 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:36.470 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.609 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:36.475 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.646 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:36.512 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.680 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:36.531 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.680 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:36.546 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.682 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:36.548 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.685 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:36.537 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.722 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:36.574 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:33.732 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:36.598 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.659 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:37.511 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.660 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:37.512 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.671 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:37.523 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.780 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:37.646 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.781 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:37.648 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.782 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:37.648 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.782 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:37.649 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.792 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:37.659 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:34.793 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:37.659 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.730 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:41.596 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.735 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:41.601 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.757 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:41.623 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.801 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:41.667 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.802 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:41.653 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.804 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:41.656 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.805 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:41.672 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.821 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:41.673 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:38.823 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:41.690 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.826 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:56.692 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.830 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:56.695 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.853 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:56.705 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.857 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:56.709 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.865 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:56.731 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.880 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:56.746 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.884 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:56.750 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.893 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:56.745 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:53.928 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:56.794 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.716 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:57.567 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.716 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:57.568 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.726 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:14:57.578 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.778 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:57.644 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.779 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:57.645 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.796 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:14:57.662 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.799 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:57.665 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.800 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:57.666 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:54:54.809 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:14:57.676 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.896 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:11.762 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.900 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:11.766 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.942 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:11.808 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.950 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:11.802 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.951 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:11.803 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.975 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:11.841 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.977 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:11.843 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:08.995 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:11.846 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:09.021 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:11.887 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.826 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:16.692 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.830 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:16.697 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.841 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:16.707 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.874 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:16.725 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.875 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:16.727 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.882 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:16.748 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.884 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:16.736 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.886 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:16.752 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:13.902 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:16.765 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.881 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:26.747 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.884 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:26.750 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.902 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:26.768 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.932 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:26.783 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.934 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:26.786 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.948 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:26.815 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.949 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:26.801 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.950 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:26.816 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:23.977 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:26.843 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.009 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:36.875 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.011 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:36.877 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.029 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:36.896 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.076 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:36.928 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.078 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:36.930 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.078 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:36.945 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.086 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:36.952 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.106 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:36.957 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:34.118 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:36.984 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.073 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:41.940 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.076 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:41.943 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.093 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:41.959 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.125 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:41.977 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.126 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:41.978 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.127 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:41.993 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.128 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:41.994 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.142 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:41.994 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:39.143 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:42.010 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.930 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:56.796 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.979 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:56.830 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.979 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:56.831 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.980 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:56.846 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.982 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:56.848 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:53.983 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:56.850 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.009 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:56.860 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.013 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:56.879 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.022 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:56.889 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.859 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:57.711 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.860 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:57.712 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.869 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:15:57.721 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.889 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:57.754 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.889 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:57.756 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.899 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:15:57.765 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.958 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:57.824 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.960 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:57.826 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:55:54.973 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:15:57.839 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.148 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:12.013 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.199 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:12.065 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.207 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:12.059 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.212 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:12.064 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.242 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:12.109 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.249 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:12.101 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.250 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:12.116 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.252 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:12.118 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:09.286 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:12.152 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.239 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:17.105 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.241 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:17.108 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.259 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:17.126 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.287 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:17.139 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.288 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:17.140 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.296 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:17.163 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.298 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:17.164 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.301 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:17.153 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:14.311 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:17.177 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.387 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:26.253 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.392 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:26.258 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.427 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:26.293 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.472 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:26.323 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.475 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:26.327 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.475 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:26.341 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.479 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:26.345 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.505 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:26.356 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:23.514 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:26.380 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.427 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:27.293 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.476 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:27.343 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.477 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:27.329 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.478 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:27.330 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.479 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:27.345 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.489 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:27.340 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.490 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:27.357 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.497 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:27.364 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:24.512 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:27.378 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.483 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:37.349 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.487 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:37.353 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.512 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:37.378 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.540 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:37.392 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.542 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:37.394 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.543 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:37.410 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.545 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:37.412 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.566 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:37.418 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:34.582 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:37.448 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: seed 41235 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (78954 ms) 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-07-27T14:56:37.908 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (90878 ms) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (12431 ms) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (182263 ms total) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (211665 ms) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (30057 ms) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (24770 ms) 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-07-27T14:56:37.909 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [3] 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [4,3] 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [5,4,3] 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting some heads 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 6 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 5 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 4 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 3 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrubs... 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (214047 ms) 2024-07-27T14:56:37.910 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12244 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10209 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (10266 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10983 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11497 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10342 ms) 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-07-27T14:56:37.911 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12413 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40421 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9317 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11345 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (18100 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (11126 ms) 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-07-27T14:56:37.912 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11958 ms) 2024-07-27T14:56:39.345 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:42.211 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.399 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:42.265 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.418 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:42.284 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.449 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:42.300 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.450 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:42.302 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.464 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:42.315 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.467 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:42.333 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.468 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:42.334 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:39.485 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:42.351 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.617 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:57.483 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.622 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:57.489 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.677 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:16:57.543 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.681 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:57.533 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.683 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:57.535 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.702 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:57.568 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.708 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:57.574 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.723 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:16:57.575 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:54.763 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:16:57.629 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.501 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:02.367 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.507 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:02.374 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.520 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:02.386 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.556 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:02.422 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.558 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:02.424 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.563 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:02.415 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.566 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:02.418 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.570 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:02.436 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:56:59.578 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:02.430 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: hmm, no HitSet yet 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: ok, hit_set contains 210:602f83fe:::foo:head 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9765 ms) 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg_num = 32 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 0 ls 1722092215,0 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 1 ls 1722092215,0 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 2 ls 1722092215,0 2024-07-27T14:57:00.242 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 3 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 4 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 5 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 6 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 7 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 8 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 9 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 10 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 11 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 12 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 13 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 14 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 15 ls 1722092215,0 2024-07-27T14:57:00.243 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 16 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 17 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 18 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 19 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 20 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 21 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 22 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 23 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 24 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 25 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 26 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 27 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 28 ls 1722092215,0 2024-07-27T14:57:00.244 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 29 ls 1722092215,0 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 30 ls 1722092215,0 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 31 ls 1722092215,0 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg_num = 32 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6cac518f:::0:head 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:02547ec2:::1:head 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f905c69b:::2:head 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:cfc208b3:::3:head 2024-07-27T14:57:00.245 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d83876eb:::4:head 2024-07-27T14:57:00.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b29083e3:::5:head 2024-07-27T14:57:00.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c4fdafeb:::6:head 2024-07-27T14:57:00.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:5c6b0b28:::7:head 2024-07-27T14:57:00.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:bd63b0f1:::8:head 2024-07-27T14:57:00.246 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e960b815:::9:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:52ea6a34:::10:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:89d3ae78:::11:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:de5d7c5f:::12:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:566253c9:::13:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:62a1935d:::14:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:863748b0:::15:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3958e169:::16:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4d4dabf9:::17:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:8391935d:::18:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:28883081:::19:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:69259c59:::20:head 2024-07-27T14:57:00.247 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4bdb80b7:::21:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a11c5d71:::22:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:271af37b:::23:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:95b121be:::24:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:58d1031b:::25:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:0a050783:::26:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c709704c:::27:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:cbe56eaf:::28:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:86b4b162:::29:head 2024-07-27T14:57:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:70d89383:::30:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:dd450c7c:::31:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6d5729b1:::32:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c388f3fb:::33:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:56cfea31:::34:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9dbc1bf7:::35:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:40b74ccd:::36:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4d5aaf42:::37:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:920f362c:::38:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6cc53222:::39:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9cad833f:::40:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:1ea84d41:::41:head 2024-07-27T14:57:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c4480ef6:::42:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a694361e:::43:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d1bd33e9:::44:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ddc2cd5d:::45:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:2b782207:::46:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7b187fca:::47:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:90ecdf6f:::48:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a5ed95fe:::49:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ea0eaa55:::50:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f33ef17b:::51:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a0d1b2f6:::52:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:60c5229e:::53:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:edcbc575:::54:head 2024-07-27T14:57:00.250 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:102cf253:::55:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:efb7fb0b:::56:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:50d0a326:::57:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d4dc5daf:::58:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3a130462:::59:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ec87ed71:::60:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d5bc9454:::61:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3ddfe313:::62:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7c2816b9:::63:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:47e00e4d:::64:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c6410c18:::65:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b48ed237:::66:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:cd63ad31:::67:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b179e92b:::68:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:0d9f741a:::69:head 2024-07-27T14:57:00.251 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6d3352ae:::70:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c6d5c19e:::71:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:bc4729c3:::72:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:77e930b9:::73:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:0abeecfd:::74:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b7c37e15:::75:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b6378398:::76:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:02bd68de:::77:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:cc795d2d:::78:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:630d4fea:::79:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e0d29ef5:::80:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:fd6f13d2:::81:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:606461d5:::82:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:eadbdc43:::83:head 2024-07-27T14:57:00.252 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:8761d0bb:::84:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9ef0186f:::85:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e0d41294:::86:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:961de695:::87:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:1423148f:::88:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:633a8fa2:::89:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a8653809:::90:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3dac8b33:::91:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:35aad435:::92:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f6dcc343:::93:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:dbbdad87:::94:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:1cb48ce0:::95:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:03cd461c:::96:head 2024-07-27T14:57:00.253 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:17a4ea99:::97:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9993c9a7:::98:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6394211c:::99:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:94c7ae57:::100:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6fdee5bb:::101:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9a477fd1:::102:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:eb850916:::103:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:affc56b9:::104:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b42dc814:::105:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f319f8f0:::106:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9a40b9de:::107:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:8b524f28:::108:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e3de589f:::109:head 2024-07-27T14:57:00.254 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:90f90a5b:::110:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a7b4f1d7:::111:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:af51766e:::112:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b6f90bd1:::113:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e0261208:::114:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c9569ef7:::115:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:61bebe50:::116:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:fe93412b:::117:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d3d38bee:::118:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3100ba0c:::119:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d0560ada:::120:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f0ea8b35:::121:head 2024-07-27T14:57:00.255 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:766f231a:::122:head 2024-07-27T14:57:00.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a07a2582:::123:head 2024-07-27T14:57:00.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:bd7c6b3a:::124:head 2024-07-27T14:57:00.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:fb2ddaff:::125:head 2024-07-27T14:57:00.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4408e1fe:::126:head 2024-07-27T14:57:00.256 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ee1df7a7:::127:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c3002909:::128:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4f48ffa9:::129:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:edf38733:::130:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c08425c0:::131:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:5f902d98:::132:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:41ea2c93:::133:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:813cee13:::134:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:0131818d:::135:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:26ba5a85:::136:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:381b8a5a:::137:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:28797e47:::138:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:bfca7f22:::139:head 2024-07-27T14:57:00.257 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:36807075:::140:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:80b03975:::141:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:5c15709b:::142:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f39ea15e:::143:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ea992956:::144:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:48887b1c:::145:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:9f24a9dd:::146:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:987f100b:::147:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d2dd3581:::148:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7fed1808:::149:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c80b70e9:::150:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:85ed90f9:::151:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:36428b24:::152:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d044c34a:::153:head 2024-07-27T14:57:00.258 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7c18bf58:::154:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d1c21232:::155:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a7a3c575:::156:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:87da0633:::157:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d5ac3822:::158:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3f20522d:::159:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6ca26563:::160:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:532ce135:::161:head 2024-07-27T14:57:00.259 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:c78863e6:::162:head 2024-07-27T14:57:09.681 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:12.546 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.683 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:12.549 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.711 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:12.577 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.744 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:12.596 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.745 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:12.597 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.750 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:12.616 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.755 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:12.618 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.763 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:12.615 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:09.776 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:12.642 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.676 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:22.542 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.679 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:22.545 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.703 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:22.569 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.729 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:22.580 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.729 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:22.596 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.730 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:22.582 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.731 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:22.597 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.755 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:22.607 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:19.764 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:22.630 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.597 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:27.463 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.603 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:27.469 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.613 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:27.480 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.656 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:27.508 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.660 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:27.512 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.670 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:27.522 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.675 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:27.541 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.676 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:27.542 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:24.686 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:27.552 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.652 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:42.518 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.654 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:42.520 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.682 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:42.548 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.704 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:42.556 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.707 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:42.559 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.710 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:42.576 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.713 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:42.580 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.746 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:42.598 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:39.753 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:42.619 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.633 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:43.485 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.634 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:43.486 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.643 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:43.495 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.706 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:43.570 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.707 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:43.571 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.718 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:43.585 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.722 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:43.589 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.723 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:43.590 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:40.733 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:43.600 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.804 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:57.669 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.806 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:57.673 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.842 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:17:57.708 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.858 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:57.710 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.859 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:57.711 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.878 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:57.744 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.883 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:57.749 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.891 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:17:57.743 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:54.921 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:17:57.787 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.721 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:02.587 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.725 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:02.592 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.740 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:02.607 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.776 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:02.628 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.777 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:02.643 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.778 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:02.644 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.779 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:02.631 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.790 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:02.656 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:57:59.795 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:02.647 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:09.087 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7462ddf6:::163:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:5d165639:::164:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f43765fc:::165:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b4c720e9:::166:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e694b040:::167:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:afa38db2:::168:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:77ba9f53:::169:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:87495034:::170:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7c96bf0e:::171:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:dbe346cc:::172:head 2024-07-27T14:58:09.088 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:e943ec24:::173:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f97a9c0c:::174:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:6f26e74d:::175:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4f95e106:::176:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:0e6f2f8f:::177:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:05db05f1:::178:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:38a78d66:::179:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:d095610b:::180:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:a1a9d709:::181:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:1e5d39db:::182:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:f7df4fb9:::183:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:03a7f161:::184:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ba70721e:::185:head 2024-07-27T14:58:09.089 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:28e5662d:::186:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:973d52de:::187:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:4303eb1c:::188:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:b990b48e:::189:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:29b8165b:::190:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:3547f197:::191:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:7e260936:::192:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:1abec7b1:::193:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:10fdda93:::194:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:15817eea:::195:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:770bab57:::196:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:ed9e13e7:::197:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:71471a8f:::198:head 2024-07-27T14:58:09.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 211:10fb1d02:::199:head 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21812 ms) 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first is 1722092238 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,0 2024-07-27T14:58:09.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,1722092246,1722092247,0 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,1722092246,1722092247,0 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092238,1722092240,1722092241,1722092243,1722092244,1722092246,1722092247,0 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092241,1722092243,1722092244,1722092246,1722092247,1722092249,1722092250,0 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first now 1722092241, trimmed 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23066 ms) 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: foo0 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (15761 ms) 2024-07-27T14:58:09.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-07-27T14:58:14.789 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:17.655 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.794 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:17.660 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.822 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:17.688 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.847 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:17.713 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.847 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:17.699 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.849 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:17.701 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.849 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:17.716 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.874 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:17.726 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:14.876 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:17.742 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.800 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:22.666 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.803 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:22.670 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.833 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:22.700 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.851 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:22.717 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.853 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:22.705 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.854 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:22.706 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.869 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:22.721 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.879 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:22.746 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:19.900 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:22.766 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.786 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:32.652 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.836 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:32.702 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.839 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:32.691 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.842 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:32.694 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.857 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:32.709 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.882 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:32.748 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.910 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:32.777 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.912 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:32.778 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:29.930 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:32.796 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.410 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:42.275 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.415 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:42.281 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.452 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:42.318 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.472 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:42.324 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.476 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:42.328 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.484 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:42.350 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.494 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:42.360 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.513 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:42.365 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:39.532 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:42.398 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.260 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:43.111 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.260 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:43.112 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.269 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:43.121 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.518 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:43.384 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.519 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:43.386 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.547 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:43.413 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.621 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:43.487 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.623 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:43.489 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:40.637 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:43.503 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:49.986 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:52.852 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:49.987 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:52.853 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.011 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:18:52.877 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.036 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:52.888 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.039 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:52.906 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.040 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:52.892 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.040 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:52.907 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.066 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:18:52.917 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:58:50.067 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:18:52.934 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.025 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:02.891 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.029 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:02.895 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.057 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:02.923 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.084 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:02.936 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.091 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:02.942 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.105 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:02.971 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.111 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:02.977 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.118 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:02.969 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:00.142 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:03.005 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.006 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:07.872 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.010 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:07.877 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.024 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:07.891 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.056 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:07.907 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.057 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:07.908 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.058 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:07.924 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.059 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:07.926 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.070 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:07.922 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:05.075 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:07.942 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.050 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:22.915 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.054 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:22.920 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.075 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:22.942 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.103 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:22.955 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.104 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:22.956 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.107 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:22.973 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.108 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:22.974 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.132 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:22.983 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:20.132 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:22.998 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.067 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:27.932 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.072 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:27.938 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.087 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:27.953 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.121 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:27.973 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.121 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:27.988 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.125 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:27.976 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.126 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:27.992 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.138 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:28.004 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:25.140 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:27.992 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.269 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:43.134 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.275 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:43.141 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.334 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:43.185 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.337 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:43.189 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.343 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:43.209 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.347 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:43.213 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.351 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:43.217 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.370 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:43.222 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:40.379 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:43.245 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.160 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:48.026 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.165 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:48.031 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.186 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:19:48.052 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.223 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:48.075 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.224 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:48.076 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.243 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:48.109 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.245 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:48.111 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.245 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:19:48.097 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T14:59:45.264 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:19:48.130 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.174 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:03.039 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.241 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:03.107 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.291 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:03.153 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.291 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:03.143 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.293 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:03.159 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.295 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:03.146 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.302 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:03.168 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.318 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:03.170 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:00.319 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:03.185 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.229 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:04.081 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.230 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:04.082 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.240 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:04.092 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.379 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:04.245 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.380 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:04.247 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.381 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:04.247 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.383 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:04.249 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.392 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:04.258 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:01.393 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:04.259 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.274 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:18.140 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.279 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:18.146 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.325 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:18.191 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.336 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:18.188 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.339 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:18.191 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.346 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:18.213 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.351 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:18.217 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.365 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:18.217 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:15.379 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:18.245 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.439 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:23.304 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.445 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:23.311 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.460 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:23.327 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.492 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:23.359 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.494 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:23.346 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.497 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:23.349 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.498 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:23.364 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.509 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:23.361 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:20.510 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:23.377 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.474 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:38.340 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.476 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:38.342 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.514 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:38.380 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.543 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:38.395 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.550 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:38.416 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.551 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:38.402 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.566 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:38.432 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.579 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:38.430 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:35.599 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:38.465 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.631 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:43.497 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.635 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:43.501 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.657 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:43.524 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.690 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:43.541 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.695 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:43.547 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.697 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:43.563 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.700 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:43.566 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.716 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:43.568 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:40.722 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:43.588 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.581 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:58.446 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.634 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:58.485 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.637 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:58.489 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.648 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:58.514 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.665 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:20:58.517 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.680 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:20:58.546 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.709 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:58.575 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.710 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:58.577 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:00:55.747 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:20:58.613 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.726 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:03.592 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.730 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:03.596 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.750 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:03.616 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.789 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:03.655 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.791 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:03.657 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.792 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:03.644 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.795 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:03.646 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.809 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:03.675 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:00.817 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:03.668 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.609 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:13.475 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.614 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:13.481 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.640 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:13.506 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.663 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:13.515 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.665 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:13.531 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.668 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:13.520 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.671 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:13.537 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.692 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:13.544 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:10.698 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:13.564 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.588 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:14.440 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.589 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:14.441 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.602 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:14.454 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.629 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:14.495 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.630 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:14.496 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.634 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:14.501 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.635 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:14.502 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.645 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:14.512 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:11.650 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:14.516 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.615 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:23.481 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.617 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:23.483 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.630 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:23.496 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.666 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:23.532 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.668 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:23.534 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.673 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:23.525 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.674 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:23.526 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.684 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:23.550 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:20.686 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:23.538 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:28.491 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T15:01:28.477+0000 2228a640 -1 snap_mapper.add_oid found existing snaps mapped on 229:a973a05b:test-rados-api-smithi087-41235-80::foo:2, removing 2024-07-27T15:01:28.507 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T15:01:28.499+0000 2f2a4640 -1 snap_mapper.add_oid found existing snaps mapped on 229:a973a05b:test-rados-api-smithi087-41235-80::foo:2, removing 2024-07-27T15:01:30.727 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:33.593 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.776 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:33.628 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.779 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:33.631 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.793 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:33.645 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.796 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:33.663 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.823 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:33.689 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.878 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:33.744 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.883 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:33.749 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:30.905 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:33.771 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.842 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:43.708 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.845 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:43.711 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.882 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:43.748 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.893 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:43.745 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.894 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:43.746 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.908 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:43.775 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.910 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:43.777 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.919 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:43.770 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:40.937 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:43.804 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.710 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:48.576 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.711 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:48.578 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.726 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:21:48.592 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.759 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:48.611 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.759 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:48.625 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.760 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:48.612 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.765 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:48.632 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.773 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:21:48.625 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:01:45.785 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:21:48.651 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:00.950 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:03.816 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:00.954 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:03.820 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.003 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:03.855 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.004 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:03.870 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.005 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:03.857 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.011 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:03.877 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.015 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:03.881 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.034 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:03.886 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:01.047 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:03.913 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20776 ms) 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25271 ms) 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3718 ms) 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5642 ms) 2024-07-27T15:02:04.155 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4539 ms) 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4873 ms) 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5406 ms) 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-07-27T15:02:04.156 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40879 ms) 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17898 ms) 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5843 ms) 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4473 ms) 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.157 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26473 ms) 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16663 ms) 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (7037 ms) 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5437 ms) 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrubs... 2024-07-27T15:02:04.158 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26629 ms) 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11932 ms) 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11313 ms) 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:02:04.159 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11040 ms) 2024-07-27T15:02:05.808 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:08.674 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.812 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:08.679 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.830 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:08.696 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.858 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:08.709 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.859 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:08.711 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.872 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:08.724 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.883 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:08.749 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.902 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:08.768 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:05.920 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:08.787 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.177 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:19.043 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.186 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:19.052 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.239 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:19.105 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.252 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:19.104 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.256 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:19.108 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.267 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:19.133 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.270 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:19.136 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.302 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:19.154 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:16.315 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:19.181 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.157 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:24.023 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.165 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:24.031 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.187 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:24.054 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.207 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:24.059 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.210 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:24.076 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.212 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:24.064 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.214 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:24.080 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.228 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:24.080 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:21.230 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:24.096 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.273 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:29.139 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.277 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:29.144 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.278 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:29.144 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.287 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:29.139 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.295 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:29.147 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.321 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:29.172 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.339 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:29.205 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.345 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:29.211 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:26.363 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:29.229 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.305 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:39.171 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.310 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:39.177 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.334 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:39.200 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.363 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:39.214 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.365 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:39.217 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.366 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:39.233 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.369 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:39.235 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.387 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:39.239 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:36.402 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:39.269 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.298 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:49.163 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.358 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:49.209 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.360 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:49.226 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.360 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:49.212 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.381 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:49.233 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.410 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:49.276 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.416 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:49.282 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.423 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:49.289 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:46.447 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:49.313 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.368 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:59.233 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.426 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:59.292 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.442 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:22:59.308 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.474 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:59.340 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.477 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:59.344 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.491 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:22:59.357 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.493 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:59.345 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.495 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:59.346 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:02:56.509 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:22:59.361 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.315 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:09.181 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.319 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:09.185 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.339 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:09.206 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.368 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:09.234 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.370 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:09.236 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.373 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:09.225 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.375 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:09.227 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.390 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:09.256 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:06.397 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:09.249 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:11.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T15:03:11.520+0000 2228a640 -1 snap_mapper.add_oid found existing snaps mapped on 102:05db248a:test-rados-api-smithi087-41235-97::foo:21, removing 2024-07-27T15:03:11.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T15:03:11.560+0000 2c69e640 -1 snap_mapper.add_oid found existing snaps mapped on 102:05db248a:test-rados-api-smithi087-41235-97::foo:21, removing 2024-07-27T15:03:16.368 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:19.233 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.370 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:19.236 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.389 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:19.255 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.419 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:19.285 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.419 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:19.271 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.420 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:19.286 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.423 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:19.275 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.439 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:19.306 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:16.441 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:19.293 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.561 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:29.426 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.562 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:29.429 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.590 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:29.456 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.631 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:29.482 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.631 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:29.497 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.635 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:29.500 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.640 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:29.492 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.671 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:29.537 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:26.678 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:29.526 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.423 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:34.289 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.427 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:34.294 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.443 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:34.309 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.476 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:34.342 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.476 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:34.328 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.477 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:34.344 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.478 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:34.329 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.492 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:34.344 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:31.493 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:34.359 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.262 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:36.128 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.270 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:36.136 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.282 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:36.134 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.285 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:36.151 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.288 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:36.140 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.296 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:36.163 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.312 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:36.178 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.320 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:36.172 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:33.376 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:36.242 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.589 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:49.454 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.591 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:49.457 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.626 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:23:49.493 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.652 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:49.518 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.653 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:49.520 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.687 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:23:49.553 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.688 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:49.539 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.689 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:49.541 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:03:46.734 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:23:49.585 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.659 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:04.525 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.665 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:04.531 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.709 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:04.575 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.731 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:04.597 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.733 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:04.599 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.734 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:04.585 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.735 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:04.587 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.770 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:04.636 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:01.776 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:04.628 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.499 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:09.365 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.546 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:09.412 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.561 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:09.427 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.591 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:09.442 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.592 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:09.444 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.606 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:09.457 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.613 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:09.479 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.615 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:09.481 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:06.629 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:09.495 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.678 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:19.543 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.683 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:19.549 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.710 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:19.576 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.734 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:19.600 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.735 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:19.602 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.747 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:19.599 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.748 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:19.600 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.773 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:19.639 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:16.774 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:19.626 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.923 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:29.789 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.924 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:29.791 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.925 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:29.776 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.927 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:29.779 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.955 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:29.822 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.956 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:29.808 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.977 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:29.843 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:26.978 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:29.845 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:27.005 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:29.871 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.689 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:34.555 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.690 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:34.557 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.706 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:34.572 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.734 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:34.586 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.736 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:34.588 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.739 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:34.605 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.742 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:34.608 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.753 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:34.605 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:31.757 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:34.623 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.779 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:49.645 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.785 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:49.651 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.822 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:49.688 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.833 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:49.699 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.833 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:49.685 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.835 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:49.687 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.838 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:49.704 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.869 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:49.720 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:46.873 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:49.736 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.797 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:54.663 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.799 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:54.665 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.814 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:24:54.680 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.846 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:54.697 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.846 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:54.698 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.861 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:24:54.712 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.865 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:54.731 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.866 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:54.732 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:04:51.881 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:24:54.747 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.825 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:04.691 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.826 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:04.692 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.847 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:04.713 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.880 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:04.746 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.880 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:04.732 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.881 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:04.747 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.883 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:04.735 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.901 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:04.767 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:01.901 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:04.753 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:11.944 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:14.810 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:11.947 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:14.814 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:11.964 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:14.830 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.005 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:14.871 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.007 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:14.858 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.013 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:14.865 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.014 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:14.881 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.027 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:14.879 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:12.031 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:14.898 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.083 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:24.949 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.085 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:24.951 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.111 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:24.977 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.139 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:25.005 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.145 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:25.012 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.146 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:24.998 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.150 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:25.002 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.180 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:25.046 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:22.181 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:25.033 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.099 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:34.964 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.102 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:34.968 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.137 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:35.003 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.156 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:35.007 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.157 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:35.009 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.160 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:35.026 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.161 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:35.027 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.182 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:35.034 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:32.188 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:35.054 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.929 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:39.795 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.974 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:39.826 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.976 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:39.828 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.978 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:39.844 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.980 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:39.846 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.980 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:39.846 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.991 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:39.843 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.996 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:39.862 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:36.997 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:39.863 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.688 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:42.554 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.693 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:42.560 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.709 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:42.575 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.760 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:42.612 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.765 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:42.631 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.765 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:42.631 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.767 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:42.619 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.793 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:42.659 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:39.794 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:42.645 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.110 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:54.975 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.111 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:54.963 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.112 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:54.978 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.114 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:54.966 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.143 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:54.995 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.146 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:55.012 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.157 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.023 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.165 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.031 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.195 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.061 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.919 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:55.770 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.919 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:55.771 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:52.929 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:25:55.781 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.061 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.927 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.063 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.929 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.074 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:25:55.940 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.101 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:55.967 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.102 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:55.968 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:05:53.116 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:25:55.982 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.154 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:10.019 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.158 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:10.024 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.206 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:10.072 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.208 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:10.060 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.210 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:10.062 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.227 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:10.093 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.231 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:10.097 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.241 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:10.093 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:07.264 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:10.130 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:08.211 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11274 ms) 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8323 ms) 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6801 ms) 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.212 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26881 ms) 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15180 ms) 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13450 ms) 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000010 69 73 |is| 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000012 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12825 ms) 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = squid 2024-07-27T15:06:08.213 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6799 ms) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (1088541 ms total) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1609 ms) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10805 ms) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19510 ms) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (14 ms) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31938 ms total) 2024-07-27T15:06:08.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8843 ms) 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10029 ms) 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrub... 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27713 ms) 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10000 ms) 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10833 ms) 2024-07-27T15:06:08.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-07-27T15:06:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10031 ms) 2024-07-27T15:06:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-07-27T15:06:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11663 ms) 2024-07-27T15:06:08.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-07-27T15:06:12.168 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:15.034 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.171 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:15.037 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.185 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:15.052 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.216 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:15.083 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.218 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:15.070 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.219 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:15.071 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.220 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:15.086 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.233 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:15.085 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:12.234 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:15.101 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.191 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:25.056 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.244 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:25.110 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.249 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:25.101 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.254 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:25.105 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.262 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:25.129 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.274 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:25.125 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.291 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:25.157 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.293 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:25.159 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:22.320 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:25.187 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.455 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:35.321 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.458 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:35.325 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.482 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:35.348 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.501 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:35.353 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.505 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:35.356 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.517 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:35.382 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.520 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:35.386 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.529 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:35.381 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:32.545 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:35.411 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.402 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:40.267 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.403 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:40.270 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.415 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:40.266 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.419 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:40.271 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.422 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:40.288 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.435 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:40.287 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.461 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:40.328 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.463 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:40.329 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:37.478 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:40.344 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.345 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:55.210 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.349 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:55.216 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.397 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:55.248 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.398 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:55.249 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.420 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:55.286 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.421 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:55.287 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.430 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:55.282 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.435 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:55.300 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:52.467 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:55.333 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.252 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:56.103 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.252 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:56.104 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.262 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:26:56.113 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.355 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:56.221 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.357 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:56.223 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.371 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:26:56.237 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.505 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:56.371 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.507 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:56.373 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:06:53.521 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:26:56.387 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.590 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:10.455 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.595 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:10.461 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.641 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:10.507 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.651 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:10.503 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.654 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:10.506 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.691 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:10.543 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.718 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:10.584 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.719 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:10.585 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:07.772 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:10.638 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.677 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:15.542 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.678 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:15.544 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.694 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:15.561 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.722 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:15.574 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.724 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:15.576 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.729 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:15.595 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.730 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:15.596 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.739 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:15.590 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:12.747 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:15.613 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.572 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:25.438 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.577 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:25.443 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.602 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:25.468 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.632 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:25.484 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.635 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:25.487 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.642 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:25.509 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.644 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:25.510 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.658 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:25.510 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:22.668 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:25.534 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.584 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:35.449 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.588 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:35.454 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.607 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:35.473 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.638 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:35.490 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.639 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:35.505 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.640 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:35.506 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.643 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:35.495 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.664 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:35.515 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:32.665 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:35.531 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.802 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:45.667 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.805 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:45.671 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.833 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:45.699 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.874 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:45.726 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.879 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:45.731 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.885 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:45.751 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.894 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:45.760 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.913 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:45.765 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:42.925 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:45.791 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.251 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:52.117 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.254 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:52.121 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.282 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:52.148 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.308 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:52.160 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.310 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:52.162 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.312 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:52.179 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.314 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:52.180 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.332 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:52.184 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:49.338 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:52.204 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.784 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:55.650 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.788 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:55.654 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.803 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:27:55.669 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.836 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:55.688 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.840 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:55.692 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.854 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:27:55.705 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.863 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:55.730 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.867 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:55.733 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:52.886 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:27:55.752 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.785 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:00.651 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.787 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:00.653 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.810 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:00.676 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.830 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:00.682 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.833 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:00.685 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.848 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:00.699 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.853 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:00.719 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.857 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:00.723 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:07:57.873 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:00.740 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.727 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:15.592 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.729 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:15.595 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.756 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:15.622 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.779 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:15.631 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.780 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:15.632 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.797 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:15.662 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.800 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:15.666 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.806 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:15.658 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:12.836 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:15.702 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.546 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:16.397 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.547 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:16.398 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.556 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:16.407 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.574 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:16.440 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.575 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:16.441 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.585 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:16.451 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.608 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:16.474 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.609 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:16.476 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:13.619 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:16.485 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.808 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:30.673 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.812 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:30.678 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.843 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:30.709 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.862 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:30.713 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.862 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:30.729 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.864 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:30.731 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.865 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:30.717 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.895 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:30.760 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:27.899 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:30.751 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.936 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:35.787 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.937 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:35.789 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.938 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:35.804 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.943 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:35.809 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.957 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:35.808 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.963 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:35.829 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.994 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:35.860 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:32.995 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:35.862 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:33.011 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:35.877 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.854 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:45.719 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.864 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:45.730 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.891 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:28:45.757 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.904 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:45.755 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.905 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:45.757 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.908 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:45.774 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.909 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:45.776 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.920 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:28:45.771 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:42.923 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:28:45.790 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:57.952 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:00.817 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:57.955 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:00.821 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.007 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:00.873 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.012 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:00.878 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.031 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:00.897 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.031 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:00.883 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.035 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:00.887 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.073 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:00.939 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.076 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:00.927 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.978 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:01.844 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.979 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:01.845 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.988 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:01.840 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.988 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:01.854 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:58.989 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:01.841 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:59.000 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:01.851 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:59.121 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:01.987 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:59.123 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:01.989 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:08:59.133 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:01.999 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:07.752 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10149 ms) 2024-07-27T15:09:07.752 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-07-27T15:09:07.752 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (15972 ms) 2024-07-27T15:09:07.752 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10214 ms) 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15075 ms) 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10015 ms) 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9659 ms) 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: hmm, no HitSet yet 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: ok, hit_set contains 261:602f83fe:::foo:head 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9751 ms) 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-07-27T15:09:07.753 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first is 1722092847 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,1722092856,1722092857,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,1722092856,1722092857,0 2024-07-27T15:09:07.754 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092847,1722092849,1722092850,1722092853,1722092854,1722092856,1722092857,0 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1722092850,1722092853,1722092854,1722092856,1722092857,1722092859,1722092860,0 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first now 1722092850, trimmed 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22296 ms) 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: foo0 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18110 ms) 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20188 ms) 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24911 ms) 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6662 ms) 2024-07-27T15:09:07.755 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4308 ms) 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5476 ms) 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3536 ms) 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (275436 ms total) 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1601006 ms total) 2024-07-27T15:09:07.756 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40952 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40952 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41147 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41147 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41663 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41663 2024-07-27T15:09:07.759 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41420 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41420 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41276 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41276 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41036 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41036 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41332 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41332 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.760 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41754 2024-07-27T15:09:07.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41754 2024-07-27T15:09:07.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:09:07.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41791 2024-07-27T15:09:07.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41791 2024-07-27T15:09:13.101 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:15.967 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.106 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:15.969 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.107 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:15.956 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.107 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:15.959 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.142 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:15.994 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.143 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:16.009 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.156 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:16.022 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.157 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:16.023 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:13.193 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:16.059 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.068 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:20.934 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.116 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:20.982 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.131 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:20.997 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.166 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:21.017 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.167 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:21.019 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.175 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:21.041 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.176 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:21.042 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.179 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:21.031 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:18.187 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:21.053 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.143 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:36.008 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.147 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:36.013 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.162 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:36.028 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.197 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:36.063 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.198 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:36.050 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.199 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:36.051 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.201 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:36.067 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.215 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:36.067 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:33.217 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:36.083 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.158 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:41.024 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.160 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:41.026 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.171 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:41.037 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.211 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:41.062 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.212 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:41.063 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.222 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:41.074 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.232 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:41.098 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.233 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:41.099 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:38.243 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:41.109 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.190 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:56.056 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.194 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:56.060 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.210 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:29:56.075 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.245 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:56.096 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.246 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:56.112 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.247 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:56.113 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.248 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:56.099 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.262 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:29:56.128 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:53.264 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:29:56.116 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.157 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:01.023 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.207 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:01.072 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.221 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:01.087 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.261 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:01.112 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.263 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:01.115 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.275 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:01.126 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.284 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:01.150 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.288 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:01.154 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:09:58.301 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:01.167 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.241 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:06.107 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.245 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:06.112 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.258 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:06.124 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.292 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:06.144 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.296 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:06.162 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.296 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:06.148 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.297 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:06.163 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.308 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:06.160 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:03.310 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:06.176 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.255 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:16.120 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.258 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:16.124 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.271 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:16.137 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.304 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:16.156 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.305 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:16.170 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.306 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:16.172 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.307 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:16.159 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.319 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:16.185 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:13.320 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:16.172 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.269 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:21.134 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.273 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:21.139 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.286 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:21.152 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.320 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:21.171 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.323 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:21.174 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.326 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:21.192 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.329 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:21.195 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.333 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:21.185 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:18.340 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:21.206 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.303 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:36.168 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.304 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:36.170 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.323 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:36.189 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.351 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:36.202 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.354 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:36.206 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.356 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:36.222 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.357 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:36.223 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.373 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:36.224 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:33.377 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:36.243 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.539 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:40.405 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.540 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:40.406 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.556 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:40.422 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.592 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:40.444 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.594 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:40.445 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.610 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:40.461 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.628 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:40.489 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.628 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:40.490 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:37.639 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:40.505 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.275 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:41.141 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.322 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:41.186 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.323 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:41.172 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.323 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:41.188 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.326 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:41.178 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.333 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:41.199 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.376 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:41.242 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.381 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:41.232 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:38.385 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:41.251 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.351 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:56.216 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.353 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:56.219 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.371 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:30:56.237 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.406 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:56.272 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.407 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:56.258 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.408 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:56.273 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.409 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:56.261 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.424 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:30:56.290 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:53.426 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:30:56.277 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.356 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.221 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.404 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.270 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.420 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.287 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.459 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.325 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.460 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.312 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.461 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.327 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.462 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.313 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.476 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.342 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:58.477 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.328 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.031 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.883 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.032 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.884 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.039 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.905 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.040 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:01.892 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.040 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.906 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.051 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:01.917 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.061 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.927 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.062 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.929 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:10:59.071 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:01.937 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.430 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:16.296 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.434 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:16.300 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.451 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:16.317 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.484 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:16.350 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.485 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:16.352 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.488 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:16.339 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.489 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:16.341 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.502 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:16.368 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:13.509 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:16.361 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.445 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:21.311 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.449 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:21.315 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.459 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:21.325 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.495 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:21.361 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.496 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:21.362 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.498 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:21.349 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.500 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:21.351 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.506 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:21.372 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:18.509 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:21.361 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.484 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:36.349 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.484 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:36.351 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.502 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:36.368 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.538 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:36.389 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.539 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:36.390 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.556 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:36.407 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.556 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:36.422 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.558 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:36.424 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:33.577 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:36.443 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.462 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:41.328 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.509 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:41.375 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.523 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:41.389 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.560 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:41.411 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.561 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:41.427 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.561 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:41.413 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.562 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:41.428 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.572 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:41.423 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:38.572 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:41.438 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.540 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:56.405 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.543 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:56.409 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.560 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:31:56.426 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.596 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:56.462 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.597 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:56.448 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.597 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:56.463 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.598 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:56.449 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.615 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:31:56.467 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:53.615 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:31:56.481 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.552 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:01.418 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.556 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:01.422 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.567 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:01.433 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.603 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:01.454 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.604 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:01.455 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.604 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:01.471 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.607 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:01.473 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.613 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:01.465 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:11:58.617 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:01.483 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.603 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:21.468 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.604 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:21.470 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.623 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:21.488 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.653 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:21.519 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.654 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:21.505 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.655 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:21.507 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.656 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:21.522 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.675 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:21.527 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:18.677 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:21.543 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.310 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:22.162 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.311 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:22.176 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.311 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:22.163 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.311 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:22.177 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.320 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:22.171 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.320 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:22.186 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.339 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:22.205 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.340 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:22.206 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:19.349 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:22.215 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.657 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:41.522 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.659 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:41.525 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.675 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:41.541 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.704 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:41.556 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.705 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:41.571 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.705 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:41.557 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.709 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:41.575 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.723 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:41.575 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:38.727 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:41.593 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.358 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:42.209 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.359 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:42.210 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.367 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:42.219 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.390 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:42.256 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.391 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:42.257 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.400 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:42.266 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.414 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:42.280 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.415 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:42.282 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:39.424 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:42.290 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: Running main() from gmock_main.cc 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [----------] Global test environment set-up. 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-07-27T15:12:48.850 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (10011 ms) 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1811713 ms) 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1821724 ms total) 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [----------] Global test environment tear-down 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1821724 ms total) 2024-07-27T15:12:48.851 INFO:tasks.workunit.client.0.smithi087.stdout: ec_io: [ PASSED ] 2 tests. 2024-07-27T15:12:48.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41062 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41062 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41696 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41696 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41565 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41565 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40927 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40927 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40991 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40991 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41631 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41631 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40918 2024-07-27T15:12:48.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40918 2024-07-27T15:12:53.935 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:56.801 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:53.939 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:56.805 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:53.971 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:32:56.837 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:53.990 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:56.856 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:53.992 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:56.857 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:54.010 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:56.861 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:54.012 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:56.863 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:54.019 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:32:56.885 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:54.044 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:32:56.895 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:58.919 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:01.784 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:58.982 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:01.848 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.001 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:01.867 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.039 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:01.890 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.040 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:01.892 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.055 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:01.920 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.056 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:01.907 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.056 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:01.922 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:12:59.084 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:01.949 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:08.720 INFO:tasks.workunit.client.0.smithi087.stdout: misc: Running main() from gmock_main.cc 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [----------] Global test environment set-up. 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.Version (4455 ms) 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4335 ms) 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.LongName (1805531 ms) 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3352 ms) 2024-07-27T15:13:08.721 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3322 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2784 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.Exec (2948 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3268 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2998 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3074 ms) 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-07-27T15:13:08.722 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2075 ms) 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3475 ms) 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [----------] 12 tests from NeoRadosMisc (1841617 ms total) 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [----------] Global test environment tear-down 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [==========] 12 tests from 1 test suite ran. (1841617 ms total) 2024-07-27T15:13:08.723 INFO:tasks.workunit.client.0.smithi087.stdout: misc: [ PASSED ] 12 tests. 2024-07-27T15:13:09.007 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:11.873 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.014 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:11.880 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.052 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:11.916 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.079 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:11.931 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.081 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:11.932 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.085 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:11.951 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.088 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:11.954 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.111 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:11.962 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:09.121 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:11.987 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.871 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:16.737 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.878 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:16.744 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.894 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:16.760 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.927 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:16.778 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.929 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:16.780 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.943 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:16.794 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.944 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:16.810 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.945 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:16.811 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:13.960 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:16.826 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.061 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:21.926 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.074 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:21.940 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.078 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:21.929 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.080 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:21.946 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.081 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:21.932 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.084 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:21.948 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.167 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:22.018 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.206 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:22.071 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:19.216 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:22.082 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:23.985 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:26.851 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:23.988 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:26.854 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.001 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:26.868 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.032 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:26.884 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.034 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:26.885 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.041 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:26.907 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.042 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:26.908 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.046 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:26.898 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:24.055 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:26.921 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.392 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:38.258 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.399 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:38.265 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.448 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:38.313 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.477 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:38.329 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.485 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:38.351 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.488 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:38.339 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.494 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:38.358 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.521 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:38.387 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:35.527 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:38.378 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.039 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:41.905 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.041 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:41.907 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.043 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:41.894 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.045 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:41.896 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.056 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:41.922 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.058 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:41.910 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.087 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:41.953 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.093 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:41.959 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.110 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:41.976 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.671 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:42.522 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.672 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:42.524 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.682 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:42.534 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.869 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:42.735 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.870 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:42.736 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.888 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:42.754 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.892 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:42.758 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.894 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:42.760 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:39.943 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:42.809 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.098 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:56.963 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.100 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:56.966 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.128 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:33:56.994 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.153 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:57.005 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.154 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:57.006 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.164 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:57.030 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.169 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:57.035 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.182 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:33:57.032 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:54.196 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:33:57.062 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.228 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:02.094 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.290 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:02.156 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.294 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:02.145 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.297 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:02.149 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.315 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:02.181 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.319 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:02.171 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.325 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:02.191 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.360 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:02.226 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:13:59.383 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:02.248 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.125 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:11.991 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.176 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:12.028 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.178 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:12.030 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.184 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:12.050 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.204 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:12.056 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.211 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:12.077 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.235 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:12.101 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.236 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:12.102 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:09.258 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:12.124 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.233 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:27.099 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.236 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:27.102 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.265 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:27.131 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.278 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:27.130 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.281 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:27.133 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.295 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:27.161 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.299 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:27.164 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.309 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:27.160 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:24.330 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:27.196 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.254 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:28.104 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.254 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:28.105 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.262 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:34:28.114 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.312 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:28.178 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.314 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:28.180 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.325 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:34:28.191 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.392 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:28.258 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.394 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:28.260 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:25.410 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:34:28.276 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:26.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-07-27T15:14:26.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-07-27T15:14:26.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-07-27T15:14:26.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-07-27T15:14:26.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4911 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1829564 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6278 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2581 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3225 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3640 ms) 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-07-27T15:14:26.081 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3195 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3473 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3486 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3294 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3362 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2657 ms) 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-07-27T15:14:26.082 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3455 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3565 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3401 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3447 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3261 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3514 ms) 2024-07-27T15:14:26.083 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3376 ms) 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3364 ms) 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2715 ms) 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3559 ms) 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3225 ms) 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-07-27T15:14:26.084 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3228 ms) 2024-07-27T15:14:26.085 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-07-27T15:14:26.085 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2877 ms) 2024-07-27T15:14:26.085 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-07-27T15:14:26.085 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3187 ms) 2024-07-27T15:14:26.085 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-07-27T15:14:59.368 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:02.232 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.372 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:02.238 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.431 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:02.283 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.433 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:02.285 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.439 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:02.305 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.440 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:02.307 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.449 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:02.315 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.485 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:02.336 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:14:59.497 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:02.362 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.362 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:07.227 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.364 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:07.230 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.398 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:07.264 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.413 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:07.265 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.414 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:07.280 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.416 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:07.268 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.419 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:07.285 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.431 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:07.283 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:04.434 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:07.300 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.372 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:17.237 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.424 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:17.290 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.449 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:17.315 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.481 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:17.332 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.486 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:17.338 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.488 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:17.354 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.491 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:17.357 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.511 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:17.363 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:14.530 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:17.396 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.559 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:27.424 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.567 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:27.433 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.590 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:27.456 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.614 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:27.465 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.615 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:27.466 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.617 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:27.483 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.624 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:27.490 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.638 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:27.489 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:24.658 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:27.523 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.457 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:32.322 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.458 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:32.324 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.472 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:32.338 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.502 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:32.354 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.503 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:32.355 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.506 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:32.373 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.508 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:32.374 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.517 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:32.368 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:29.522 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:32.388 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.071 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:38.937 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.104 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:38.970 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.132 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:38.998 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.134 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:38.985 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.135 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:39.001 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.171 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:39.023 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.181 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:39.047 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.206 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:39.058 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:36.207 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:39.073 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.616 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:47.481 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.619 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:47.485 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.642 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:35:47.508 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.667 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:47.533 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.670 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:47.536 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.693 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:35:47.559 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.697 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:47.548 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.700 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:47.551 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:44.722 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:35:47.573 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.547 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:02.413 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.548 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:02.414 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.576 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:02.442 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.602 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:02.453 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.603 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:02.454 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.607 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:02.473 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.609 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:02.475 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.632 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:02.483 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:15:59.644 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:02.510 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.620 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:07.485 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.622 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:07.489 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.630 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:07.482 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.633 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:07.494 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.633 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:07.496 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.634 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:07.486 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.698 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:07.564 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.710 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:07.561 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:04.710 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:07.576 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.606 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:17.472 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.610 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:17.476 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.636 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:17.502 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.659 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:17.511 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.662 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:17.514 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.675 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:17.541 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.682 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:17.548 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.685 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:17.537 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:14.709 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:17.575 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.755 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:27.620 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.756 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:27.623 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.780 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:27.645 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.802 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:27.653 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.803 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:27.655 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.807 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:27.674 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.809 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:27.675 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.825 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:27.677 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:24.834 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:27.698 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.779 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:32.644 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.828 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:32.694 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.843 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:32.709 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.874 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:32.725 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.876 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:32.728 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.878 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:32.744 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.880 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:32.746 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.891 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:32.743 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:29.895 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:32.761 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.756 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:47.621 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.758 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:47.624 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.789 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:47.655 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.808 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:47.659 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.810 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:47.662 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.816 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:47.680 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.816 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:47.682 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.838 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:47.689 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:44.846 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:47.712 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.687 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:48.539 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.688 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:48.540 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.697 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:36:48.549 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.726 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:48.592 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.727 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:48.593 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.727 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:48.594 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.729 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:48.595 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.738 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:36:48.603 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:45.739 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:36:48.605 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.792 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:02.657 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.796 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:02.662 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.836 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:02.702 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.843 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:02.694 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.844 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:02.695 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.863 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:02.729 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.867 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:02.733 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.869 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:02.721 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:16:59.895 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:02.759 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:02.733 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripAppendPP (3613 ms) 2024-07-27T15:17:02.733 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3244 ms) 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3511 ms) 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2477 ms) 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: setting pool EIO 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: max_success 9, min_failed 10 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6535 ms) 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3523 ms) 2024-07-27T15:17:02.734 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1938743 ms total) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3405 ms) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10054 ms) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16372 ms) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3315 ms) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33146 ms total) 2024-07-27T15:17:02.735 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3264 ms) 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3264 ms total) 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15425 ms) 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7510 ms) 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-07-27T15:17:02.736 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7176 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6940 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3548 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6745 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7960 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6948 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7840 ms) 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-07-27T15:17:02.737 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7577 ms) 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (8117 ms) 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7296 ms) 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-07-27T15:17:02.738 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7871 ms) 2024-07-27T15:17:09.878 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:12.744 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:09.941 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:12.792 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:09.943 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:12.794 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:09.943 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:12.809 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:09.971 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:12.822 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:09.978 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:12.844 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:10.009 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:12.875 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:10.014 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:12.880 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:10.045 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:12.911 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.836 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:17.702 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.840 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:17.706 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.853 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:17.719 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.887 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:17.738 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.888 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:17.739 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.888 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:17.754 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.893 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:17.759 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.900 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:17.752 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:14.913 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:17.779 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:29.947 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:32.813 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:29.952 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:32.818 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:29.960 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:32.811 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:29.962 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:32.828 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:30.017 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:32.869 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:30.018 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:32.884 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:30.048 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:32.914 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:30.051 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:32.903 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:30.064 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:32.930 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.223 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:36.089 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.225 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:36.091 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.238 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:36.104 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.238 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:36.090 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.242 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:36.094 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.245 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:36.111 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.267 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:36.133 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.320 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:36.171 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:33.334 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:36.199 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.918 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:37.784 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.919 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:37.786 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.929 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:37.795 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.966 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:37.818 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.968 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:37.820 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.970 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:37.836 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.972 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:37.838 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.977 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:37.829 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:34.981 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:37.847 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:49.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-07-27T15:17:49.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3292 ms) 2024-07-27T15:17:49.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-07-27T15:17:49.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7189 ms) 2024-07-27T15:17:49.417 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6690 ms) 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7516 ms) 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6986 ms) 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7880 ms) 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7127 ms) 2024-07-27T15:17:49.418 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (147633 ms total) 2024-07-27T15:17:49.419 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-07-27T15:17:49.419 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-07-27T15:17:49.419 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2122787 ms total) 2024-07-27T15:17:49.419 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-07-27T15:17:49.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40938 2024-07-27T15:17:49.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40938 2024-07-27T15:17:49.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41936 2024-07-27T15:17:49.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41936 2024-07-27T15:17:49.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41168 2024-07-27T15:17:49.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41168 2024-07-27T15:17:49.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42026 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 42026 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41298 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41298 2024-07-27T15:17:49.428 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41729 2024-07-27T15:17:49.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41729 2024-07-27T15:17:49.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42059 2024-07-27T15:17:49.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 42059 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41191 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41191 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41650 2024-07-27T15:17:49.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41650 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41911 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41911 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41702 2024-07-27T15:17:49.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41702 2024-07-27T15:17:49.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42000 2024-07-27T15:17:49.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 42000 2024-07-27T15:17:49.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41520 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41520 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41117 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41117 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42039 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 42039 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41439 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41439 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40911 2024-07-27T15:17:49.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40911 2024-07-27T15:17:49.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=40969 2024-07-27T15:17:49.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 40969 2024-07-27T15:17:49.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41370 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41370 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41017 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41017 2024-07-27T15:17:49.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41457 2024-07-27T15:17:49.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41457 2024-07-27T15:17:49.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41474 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41474 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=41865 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 41865 2024-07-27T15:17:49.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-07-27T15:17:49.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=42108 2024-07-27T15:17:49.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 42108 2024-07-27T15:17:49.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ exit 0 2024-07-27T15:17:49.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ cleanup 2024-07-27T15:17:49.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ pkill -P 40905 2024-07-27T15:17:49.654 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2024-07-27T15:17:49.655 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-07-27T15:17:49.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-07-27T15:17:49.683 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-07-27T15:17:49.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-07-27T15:17:49.967 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:52.833 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:49.968 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:52.834 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:49.997 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:52.862 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.016 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:52.868 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.017 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:52.883 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.018 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:52.869 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.020 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:52.886 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.027 DEBUG:teuthology.parallel:result is None 2024-07-27T15:17:50.028 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-07-27T15:17:50.045 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:52.896 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.052 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:52.918 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.053 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-07-27T15:17:50.053 DEBUG:teuthology.orchestra.run.smithi087:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-07-27T15:17:50.113 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-07-27T15:17:50.113 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-07-27T15:17:50.124 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-27T15:17:50.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T15:17:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.269+0000 7fa1d3e33640 10 monclient: get_monmap_and_config 2024-07-27T15:17:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.269+0000 7fa1d3e33640 10 monclient: build_initial_monmap 2024-07-27T15:17:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.269+0000 7fa1d3e33640 10 monclient: monmap: 2024-07-27T15:17:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:50.270392+0000 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:50.270392+0000 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:50.271 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.269+0000 7fa1d3e33640 1 Processor -- start 2024-07-27T15:17:50.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 -- start start 2024-07-27T15:17:50.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient: init 2024-07-27T15:17:50.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc0ad550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-b con 0x7fa1cc0ab6c0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1cc0aeb80 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa1cc0abd90 0x7fa1cc0ada90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-c con 0x7fa1cc0abd90 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1cc0ace50 con 0x7fa1cc0aeb80 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fa1cc0abaa0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa1cc0aabc0 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:50.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.270+0000 7fa1d3e33640 10 monclient(hunting): authenticate will time out at 3410.296836s 2024-07-27T15:17:50.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.273+0000 7fa1d13ce640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa1cc0aeb80 0x7fa1cc141910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:39370/0 (socket says 172.21.15.87:39370) 2024-07-27T15:17:50.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.273+0000 7fa1d13ce640 1 -- 172.21.15.87:0/2438333720 learned_addr learned my addr 172.21.15.87:0/2438333720 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:50.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.273+0000 7fa1d1bcf640 1 --2- 172.21.15.87:0/2438333720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc0ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:50.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.276+0000 7fa1d1bcf640 10 monclient(hunting): get_auth_request con 0x7fa1cc0ab6c0 auth_method 0 2024-07-27T15:17:50.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.276+0000 7fa1d1bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:50.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.276+0000 7fa1d1bcf640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:50.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.276+0000 7fa1d1bcf640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:50.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.278+0000 7fa1d23d0640 1 --2- 172.21.15.87:0/2438333720 >> v2:172.21.15.112:3300/0 conn(0x7fa1cc0abd90 0x7fa1cc0ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:50.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.280+0000 7fa1d1bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:50.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.280+0000 7fa1d1bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:50.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.280+0000 7fa1d1bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:50.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.281+0000 7fa1d23d0640 10 monclient(hunting): get_auth_request con 0x7fa1cc0abd90 auth_method 0 2024-07-27T15:17:50.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.281+0000 7fa1d23d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:50.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.281+0000 7fa1d23d0640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:50.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.281+0000 7fa1d23d0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 1 -- 172.21.15.87:0/2438333720 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1637876547 0 0) 0x7fa1cc0ace50 con 0x7fa1cc0aeb80 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 10 monclient(hunting): _negotiate 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 10 monclient(hunting): my global_id is 14724 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d0bcd640 1 -- 172.21.15.87:0/2438333720 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b8003520 con 0x7fa1cc0aeb80 2024-07-27T15:17:50.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 10 monclient(hunting): handle_auth_done global_id 15491 payload 274 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 1 -- 172.21.15.87:0/2438333720 >> v2:172.21.15.112:3300/0 conn(0x7fa1cc0abd90 msgr2=0x7fa1cc0ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 1 --2- 172.21.15.87:0/2438333720 >> v2:172.21.15.112:3300/0 conn(0x7fa1cc0abd90 0x7fa1cc0ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 1 -- 172.21.15.87:0/2438333720 >> v1:172.21.15.87:6789/0 conn(0x7fa1cc0aeb80 legacy=0x7fa1cc141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 10 monclient: _finish_hunting 0 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 1 monclient: found mon.noname-b 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:17:50.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.283+0000 7fa1d1bcf640 1 -- 172.21.15.87:0/2438333720 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1cc0ad130 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.288+0000 7fa1d1bcf640 1 --2- 172.21.15.87:0/2438333720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc0ad550 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7fa1c800a940 tx=0x7fa1c802ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be815451bf8b264e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:50.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 1 -- 172.21.15.87:0/2438333720 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c803a070 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: dump: 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: _finish_auth 0 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 1 -- 172.21.15.87:0/2438333720 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c802dbc0 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d3e33640 5 monclient: authenticate success, global_id 15491 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d3e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d3e33640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d3e33640 4 set_mon_vals no callback set 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 1 -- 172.21.15.87:0/2438333720 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c800b040 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.305+0000 7fa1d0bcd640 10 monclient: dump: 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:50.309 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 10 monclient: shutdown 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2438333720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 msgr2=0x7fa1cc0ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2438333720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc0ad550 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7fa1c800a940 tx=0x7fa1c802ce10 comp rx=0 tx=0).stop 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2438333720 shutdown_connections 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2438333720 >> v2:172.21.15.112:3300/0 conn(0x7fa1cc0abd90 0x7fa1cc0ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2438333720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc0ad550 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2438333720 >> 172.21.15.87:0/2438333720 conn(0x7fa1cc067f60 msgr2=0x7fa1cc0b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2438333720 shutdown_connections 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.306+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2438333720 wait complete. 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.307+0000 7fa1d3e33640 10 monclient: build_initial_monmap 2024-07-27T15:17:50.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.307+0000 7fa1d3e33640 10 monclient: monmap: 2024-07-27T15:17:50.313 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:50.313 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 librados: starting msgr at 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 librados: starting objecter 2024-07-27T15:17:50.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 Processor -- start 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 -- start start 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 librados: setting wanted keys 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 librados: calling monclient init 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient: init 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc11d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-c con 0x7fa1cc0ab6c0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-a con 0x7fa1cc0abd90 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1cc0aeb80 0x7fa1cc13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): picked mon.noname-b con 0x7fa1cc0aeb80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc142270 con 0x7fa1cc0abd90 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc0af090 con 0x7fa1cc0aeb80 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.308+0000 7fa1d3e33640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa1cc11e9c0 con 0x7fa1cc0ab6c0 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.309+0000 7fa1d3e33640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.309+0000 7fa1d3e33640 10 monclient(hunting): authenticate will time out at 3410.335013s 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.311+0000 7fa1d13ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37382/0 (socket says 172.21.15.87:37382) 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 1 -- 172.21.15.87:0/2218131862 learned_addr learned my addr 172.21.15.87:0/2218131862 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 10 monclient(hunting): get_auth_request con 0x7fa1cc0abd90 auth_method 0 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.313+0000 7fa1d13ce640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:50.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.315+0000 7fa1d13ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:50.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.315+0000 7fa1d13ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:50.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.315+0000 7fa1d13ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 10 monclient(hunting): handle_auth_done global_id 14727 payload 824 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 -- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 msgr2=0x7fa1cc11d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc11d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 -- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1cc0aeb80 msgr2=0x7fa1cc13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1cc0aeb80 0x7fa1cc13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:50.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 10 monclient: _finish_hunting 0 2024-07-27T15:17:50.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 monclient: found mon.noname-a 2024-07-27T15:17:50.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:50.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:50.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.318+0000 7fa1d13ce640 1 -- 172.21.15.87:0/2218131862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1cc1435f0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.336+0000 7fa1d13ce640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc001eb0 tx=0x7fa1bc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:50.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.336+0000 7fa1c27fc640 10 client.?.objecter ms_handle_connect 0x7fa1cc0abd90 2024-07-27T15:17:50.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.336+0000 7fa1c27fc640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc017070 con 0x7fa1cc0abd90 2024-07-27T15:17:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: dump: 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: _finish_auth 0 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: _check_auth_tickets 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1bc005d00 con 0x7fa1cc0abd90 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 5 monclient: authenticate success, global_id 14727 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc007310 con 0x7fa1cc0abd90 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 10 monclient: _renew_subs 2024-07-27T15:17:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1cc11cff0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c3fff640 4 set_mon_vals no callback set 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1c27fc640 10 monclient: dump: 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:50.386 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 10 client.14727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 10 monclient: _renew_subs 2024-07-27T15:17:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1cc13f8c0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.380+0000 7fa1d3e33640 1 librados: init done 2024-07-27T15:17:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.381+0000 7fa1d3e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.381+0000 7fa1d3e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.381+0000 7fa1d3e33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.381+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1cc0af4f0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.797+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa1bc01a070 con 0x7fa1cc0abd90 2024-07-27T15:17:50.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.798+0000 7fa1c27fc640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 0x7fa1b006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:50.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.798+0000 7fa1d1bcf640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 0x7fa1b006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:50.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.798+0000 7fa1d1bcf640 10 monclient: get_auth_request con 0x7fa1b006d810 auth_method 0 2024-07-27T15:17:50.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.799+0000 7fa1c27fc640 10 client.14727.objecter ms_handle_connect 0x7fa1b006d810 2024-07-27T15:17:50.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.799+0000 7fa1d1bcf640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 0x7fa1b006fcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8037000 tx=0x7fa1c802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7fa1bc056ab0 con 0x7fa1cc0abd90 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 10 client.14727.objecter ms_dispatch 0x7fa1cc000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 3 client.14727.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 3 client.14727.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 20 client.14727.objecter dump_active .. 0 homeless 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fa1bc021020 con 0x7fa1cc0abd90 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:50.825+0000 7fa1c27fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:50.830 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:53.681 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.831 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:53.682 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.839 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:37:53.690 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.917 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:53.783 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.918 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:53.785 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:50.930 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:37:53.796 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:51.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.021+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1cc11d220 con 0x7fa1b006d810 2024-07-27T15:17:51.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.025+0000 7fa1c27fc640 1 -- 172.21.15.87:0/2218131862 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116367 (secure 0 0 0) 0x7fa1cc11d220 con 0x7fa1b006d810 2024-07-27T15:17:51.027 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:17:51.028 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 10 librados: watch_flush enter 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 10 librados: watch_flush exit 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 20 client.14727.objecter shutdown clearing up homeless session... 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 10 client.14727.objecter successfully canceled tick 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 msgr2=0x7fa1b006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 0x7fa1b006fcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8037000 tx=0x7fa1c802f000 comp rx=0 tx=0).stop 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 10 monclient: shutdown 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 msgr2=0x7fa1cc11cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc001eb0 tx=0x7fa1bc00c820 comp rx=0 tx=0).stop 2024-07-27T15:17:51.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 shutdown_connections 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa1b006d810 0x7fa1b006fcd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1cc0aeb80 0x7fa1cc13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1cc0abd90 0x7fa1cc11cab0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 --2- 172.21.15.87:0/2218131862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1cc0ab6c0 0x7fa1cc11d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 >> 172.21.15.87:0/2218131862 conn(0x7fa1cc067f60 msgr2=0x7fa1cc0b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 shutdown_connections 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 -- 172.21.15.87:0/2218131862 wait complete. 2024-07-27T15:17:51.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.027+0000 7fa1d3e33640 1 librados: shutdown 2024-07-27T15:17:51.040 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":2577,"stamp":"2024-07-27T15:17:50.493597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":122,"num_write_kb":1381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595096,"kb_used_data":44392,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754379624,"statfs":{"total":773094113280,"available":772484734976,"internally_reserved":0,"allocated":45457408,"data_stored":21951472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563793456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":182,"apply_latency_ms":182,"commit_latency_ns":182000000,"apply_latency_ns":182000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.563109"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":2814,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.707879+0000","last_change":"2024-07-27T14:42:41.580179+0000","last_active":"2024-07-27T15:17:47.707879+0000","last_peered":"2024-07-27T15:17:47.707879+0000","last_clean":"2024-07-27T15:17:47.707879+0000","last_became_active":"2024-07-27T14:42:41.574286+0000","last_became_peered":"2024-07-27T14:42:41.574286+0000","last_unstale":"2024-07-27T15:17:47.707879+0000","last_undegraded":"2024-07-27T15:17:47.707879+0000","last_fullsized":"2024-07-27T15:17:47.707879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:03:00.545391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":2822,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.768072+0000","last_change":"2024-07-27T14:42:33.146971+0000","last_active":"2024-07-27T15:17:47.768072+0000","last_peered":"2024-07-27T15:17:47.768072+0000","last_clean":"2024-07-27T15:17:47.768072+0000","last_became_active":"2024-07-27T14:42:33.130123+0000","last_became_peered":"2024-07-27T14:42:33.130123+0000","last_unstale":"2024-07-27T15:17:47.768072+0000","last_undegraded":"2024-07-27T15:17:47.768072+0000","last_fullsized":"2024-07-27T15:17:47.768072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:20:51.941325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2826,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.956847+0000","last_change":"2024-07-27T14:42:32.205230+0000","last_active":"2024-07-27T15:17:45.956847+0000","last_peered":"2024-07-27T15:17:45.956847+0000","last_clean":"2024-07-27T15:17:45.956847+0000","last_became_active":"2024-07-27T14:42:32.196113+0000","last_became_peered":"2024-07-27T14:42:32.196113+0000","last_unstale":"2024-07-27T15:17:45.956847+0000","last_undegraded":"2024-07-27T15:17:45.956847+0000","last_fullsized":"2024-07-27T15:17:45.956847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:11:27.161409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2825,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.975445+0000","last_change":"2024-07-27T14:42:33.556536+0000","last_active":"2024-07-27T15:17:45.975445+0000","last_peered":"2024-07-27T15:17:45.975445+0000","last_clean":"2024-07-27T15:17:45.975445+0000","last_became_active":"2024-07-27T14:42:33.550008+0000","last_became_peered":"2024-07-27T14:42:33.550008+0000","last_unstale":"2024-07-27T15:17:45.975445+0000","last_undegraded":"2024-07-27T15:17:45.975445+0000","last_fullsized":"2024-07-27T15:17:45.975445+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:19:45.505942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":2827,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.697983+0000","last_change":"2024-07-27T14:42:33.671344+0000","last_active":"2024-07-27T15:17:47.697983+0000","last_peered":"2024-07-27T15:17:47.697983+0000","last_clean":"2024-07-27T15:17:47.697983+0000","last_became_active":"2024-07-27T14:42:33.554937+0000","last_became_peered":"2024-07-27T14:42:33.554937+0000","last_unstale":"2024-07-27T15:17:47.697983+0000","last_undegraded":"2024-07-27T15:17:47.697983+0000","last_fullsized":"2024-07-27T15:17:47.697983+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:25:20.740493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2802,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.100193+0000","last_change":"2024-07-27T14:42:43.884287+0000","last_active":"2024-07-27T15:17:48.100193+0000","last_peered":"2024-07-27T15:17:48.100193+0000","last_clean":"2024-07-27T15:17:48.100193+0000","last_became_active":"2024-07-27T14:42:43.877378+0000","last_became_peered":"2024-07-27T14:42:43.877378+0000","last_unstale":"2024-07-27T15:17:48.100193+0000","last_undegraded":"2024-07-27T15:17:48.100193+0000","last_fullsized":"2024-07-27T15:17:48.100193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:37:47.388561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":2817,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.369035+0000","last_change":"2024-07-27T14:42:35.229319+0000","last_active":"2024-07-27T15:17:49.369035+0000","last_peered":"2024-07-27T15:17:49.369035+0000","last_clean":"2024-07-27T15:17:49.369035+0000","last_became_active":"2024-07-27T14:42:35.220195+0000","last_became_peered":"2024-07-27T14:42:35.220195+0000","last_unstale":"2024-07-27T15:17:49.369035+0000","last_undegraded":"2024-07-27T15:17:49.369035+0000","last_fullsized":"2024-07-27T15:17:49.369035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:46:10.403748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.946537+0000","last_change":"2024-07-27T14:42:33.873297+0000","last_active":"2024-07-27T15:17:45.946537+0000","last_peered":"2024-07-27T15:17:45.946537+0000","last_clean":"2024-07-27T15:17:45.946537+0000","last_became_active":"2024-07-27T14:42:33.867602+0000","last_became_peered":"2024-07-27T14:42:33.867602+0000","last_unstale":"2024-07-27T15:17:45.946537+0000","last_undegraded":"2024-07-27T15:17:45.946537+0000","last_fullsized":"2024-07-27T15:17:45.946537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:17:09.550687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.488131+0000","last_change":"2024-07-27T14:42:34.920701+0000","last_active":"2024-07-27T15:17:48.488131+0000","last_peered":"2024-07-27T15:17:48.488131+0000","last_clean":"2024-07-27T15:17:48.488131+0000","last_became_active":"2024-07-27T14:42:34.746905+0000","last_became_peered":"2024-07-27T14:42:34.746905+0000","last_unstale":"2024-07-27T15:17:48.488131+0000","last_undegraded":"2024-07-27T15:17:48.488131+0000","last_fullsized":"2024-07-27T15:17:48.488131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:30:00.579219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.766173+0000","last_change":"2024-07-27T14:42:37.296338+0000","last_active":"2024-07-27T15:17:47.766173+0000","last_peered":"2024-07-27T15:17:47.766173+0000","last_clean":"2024-07-27T15:17:47.766173+0000","last_became_active":"2024-07-27T14:42:37.287685+0000","last_became_peered":"2024-07-27T14:42:37.287685+0000","last_unstale":"2024-07-27T15:17:47.766173+0000","last_undegraded":"2024-07-27T15:17:47.766173+0000","last_fullsized":"2024-07-27T15:17:47.766173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:04:06.389200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":2814,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.360888+0000","last_change":"2024-07-27T14:42:35.360457+0000","last_active":"2024-07-27T15:17:49.360888+0000","last_peered":"2024-07-27T15:17:49.360888+0000","last_clean":"2024-07-27T15:17:49.360888+0000","last_became_active":"2024-07-27T14:42:35.354552+0000","last_became_peered":"2024-07-27T14:42:35.354552+0000","last_unstale":"2024-07-27T15:17:49.360888+0000","last_undegraded":"2024-07-27T15:17:49.360888+0000","last_fullsized":"2024-07-27T15:17:49.360888+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:22:00.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.523171+0000","last_change":"2024-07-27T14:42:34.577783+0000","last_active":"2024-07-27T15:17:48.523171+0000","last_peered":"2024-07-27T15:17:48.523171+0000","last_clean":"2024-07-27T15:17:48.523171+0000","last_became_active":"2024-07-27T14:42:34.568719+0000","last_became_peered":"2024-07-27T14:42:34.568719+0000","last_unstale":"2024-07-27T15:17:48.523171+0000","last_undegraded":"2024-07-27T15:17:48.523171+0000","last_fullsized":"2024-07-27T15:17:48.523171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:14:22.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2929,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.109246+0000","last_change":"2024-07-27T14:41:03.617220+0000","last_active":"2024-07-27T15:17:48.109246+0000","last_peered":"2024-07-27T15:17:48.109246+0000","last_clean":"2024-07-27T15:17:48.109246+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T15:17:48.109246+0000","last_undegraded":"2024-07-27T15:17:48.109246+0000","last_fullsized":"2024-07-27T15:17:48.109246+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_clean_scrub_stamp":"2024-07-27T14:41:00.649818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:14:15.632432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":2847,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.738254+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T15:17:47.738254+0000","last_peered":"2024-07-27T15:17:47.738254+0000","last_clean":"2024-07-27T15:17:47.738254+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T15:17:47.738254+0000","last_undegraded":"2024-07-27T15:17:47.738254+0000","last_fullsized":"2024-07-27T15:17:47.738254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:40:29.539136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":2825,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.769897+0000","last_change":"2024-07-27T14:42:37.311409+0000","last_active":"2024-07-27T15:17:47.769897+0000","last_peered":"2024-07-27T15:17:47.769897+0000","last_clean":"2024-07-27T15:17:47.769897+0000","last_became_active":"2024-07-27T14:42:37.189428+0000","last_became_peered":"2024-07-27T14:42:37.189428+0000","last_unstale":"2024-07-27T15:17:47.769897+0000","last_undegraded":"2024-07-27T15:17:47.769897+0000","last_fullsized":"2024-07-27T15:17:47.769897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:15:12.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2846,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.695050+0000","last_change":"2024-07-27T14:41:21.948784+0000","last_active":"2024-07-27T15:17:47.695050+0000","last_peered":"2024-07-27T15:17:47.695050+0000","last_clean":"2024-07-27T15:17:47.695050+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T15:17:47.695050+0000","last_undegraded":"2024-07-27T15:17:47.695050+0000","last_fullsized":"2024-07-27T15:17:47.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:23:48.411987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.959864+0000","last_change":"2024-07-27T14:42:32.948619+0000","last_active":"2024-07-27T15:17:45.959864+0000","last_peered":"2024-07-27T15:17:45.959864+0000","last_clean":"2024-07-27T15:17:45.959864+0000","last_became_active":"2024-07-27T14:42:32.825779+0000","last_became_peered":"2024-07-27T14:42:32.825779+0000","last_unstale":"2024-07-27T15:17:45.959864+0000","last_undegraded":"2024-07-27T15:17:45.959864+0000","last_fullsized":"2024-07-27T15:17:45.959864+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:44:30.813522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.506184+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T15:17:48.506184+0000","last_peered":"2024-07-27T15:17:48.506184+0000","last_clean":"2024-07-27T15:17:48.506184+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T15:17:48.506184+0000","last_undegraded":"2024-07-27T15:17:48.506184+0000","last_fullsized":"2024-07-27T15:17:48.506184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2813,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.720308+0000","last_change":"2024-07-27T14:43:03.368313+0000","last_active":"2024-07-27T15:17:47.720308+0000","last_peered":"2024-07-27T15:17:47.720308+0000","last_clean":"2024-07-27T15:17:47.720308+0000","last_became_active":"2024-07-27T14:43:03.360033+0000","last_became_peered":"2024-07-27T14:43:03.360033+0000","last_unstale":"2024-07-27T15:17:47.720308+0000","last_undegraded":"2024-07-27T15:17:47.720308+0000","last_fullsized":"2024-07-27T15:17:47.720308+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:21:29.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":2848,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.678919+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T15:17:47.678919+0000","last_peered":"2024-07-27T15:17:47.678919+0000","last_clean":"2024-07-27T15:17:47.678919+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T15:17:47.678919+0000","last_undegraded":"2024-07-27T15:17:47.678919+0000","last_fullsized":"2024-07-27T15:17:47.678919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:39:58.978046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.704667+0000","last_change":"2024-07-27T14:42:33.420873+0000","last_active":"2024-07-27T15:17:47.704667+0000","last_peered":"2024-07-27T15:17:47.704667+0000","last_clean":"2024-07-27T15:17:47.704667+0000","last_became_active":"2024-07-27T14:42:33.415149+0000","last_became_peered":"2024-07-27T14:42:33.415149+0000","last_unstale":"2024-07-27T15:17:47.704667+0000","last_undegraded":"2024-07-27T15:17:47.704667+0000","last_fullsized":"2024-07-27T15:17:47.704667+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:45:18.005668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2848,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.951360+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T15:17:45.951360+0000","last_peered":"2024-07-27T15:17:45.951360+0000","last_clean":"2024-07-27T15:17:45.951360+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T15:17:45.951360+0000","last_undegraded":"2024-07-27T15:17:45.951360+0000","last_fullsized":"2024-07-27T15:17:45.951360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.698776+0000","last_change":"2024-07-27T14:42:57.118705+0000","last_active":"2024-07-27T15:17:47.698776+0000","last_peered":"2024-07-27T15:17:47.698776+0000","last_clean":"2024-07-27T15:17:47.698776+0000","last_became_active":"2024-07-27T14:42:57.112921+0000","last_became_peered":"2024-07-27T14:42:57.112921+0000","last_unstale":"2024-07-27T15:17:47.698776+0000","last_undegraded":"2024-07-27T15:17:47.698776+0000","last_fullsized":"2024-07-27T15:17:47.698776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:02:08.809341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2843,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.771806+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T15:17:47.771806+0000","last_peered":"2024-07-27T15:17:47.771806+0000","last_clean":"2024-07-27T15:17:47.771806+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T15:17:47.771806+0000","last_undegraded":"2024-07-27T15:17:47.771806+0000","last_fullsized":"2024-07-27T15:17:47.771806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:57:17.520382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.702273+0000","last_change":"2024-07-27T14:42:36.978696+0000","last_active":"2024-07-27T15:17:47.702273+0000","last_peered":"2024-07-27T15:17:47.702273+0000","last_clean":"2024-07-27T15:17:47.702273+0000","last_became_active":"2024-07-27T14:42:36.941428+0000","last_became_peered":"2024-07-27T14:42:36.941428+0000","last_unstale":"2024-07-27T15:17:47.702273+0000","last_undegraded":"2024-07-27T15:17:47.702273+0000","last_fullsized":"2024-07-27T15:17:47.702273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:34:55.684245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2835,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.063984+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T15:17:46.063984+0000","last_peered":"2024-07-27T15:17:46.063984+0000","last_clean":"2024-07-27T15:17:46.063984+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T15:17:46.063984+0000","last_undegraded":"2024-07-27T15:17:46.063984+0000","last_fullsized":"2024-07-27T15:17:46.063984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2819,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.010815+0000","last_change":"2024-07-27T14:42:32.399654+0000","last_active":"2024-07-27T15:17:46.010815+0000","last_peered":"2024-07-27T15:17:46.010815+0000","last_clean":"2024-07-27T15:17:46.010815+0000","last_became_active":"2024-07-27T14:42:32.329977+0000","last_became_peered":"2024-07-27T14:42:32.329977+0000","last_unstale":"2024-07-27T15:17:46.010815+0000","last_undegraded":"2024-07-27T15:17:46.010815+0000","last_fullsized":"2024-07-27T15:17:46.010815+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:17:59.414451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2840,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.777392+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T15:17:47.777392+0000","last_peered":"2024-07-27T15:17:47.777392+0000","last_clean":"2024-07-27T15:17:47.777392+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T15:17:47.777392+0000","last_undegraded":"2024-07-27T15:17:47.777392+0000","last_fullsized":"2024-07-27T15:17:47.777392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:05:24.194010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"44'7","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.354492+0000","last_change":"2024-07-27T14:42:34.811194+0000","last_active":"2024-07-27T15:17:49.354492+0000","last_peered":"2024-07-27T15:17:49.354492+0000","last_clean":"2024-07-27T15:17:49.354492+0000","last_became_active":"2024-07-27T14:42:34.791352+0000","last_became_peered":"2024-07-27T14:42:34.791352+0000","last_unstale":"2024-07-27T15:17:49.354492+0000","last_undegraded":"2024-07-27T15:17:49.354492+0000","last_fullsized":"2024-07-27T15:17:49.354492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:36:47.747804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":2816,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.358672+0000","last_change":"2024-07-27T14:42:35.255013+0000","last_active":"2024-07-27T15:17:49.358672+0000","last_peered":"2024-07-27T15:17:49.358672+0000","last_clean":"2024-07-27T15:17:49.358672+0000","last_became_active":"2024-07-27T14:42:35.248761+0000","last_became_peered":"2024-07-27T14:42:35.248761+0000","last_unstale":"2024-07-27T15:17:49.358672+0000","last_undegraded":"2024-07-27T15:17:49.358672+0000","last_fullsized":"2024-07-27T15:17:49.358672+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:43:59.817053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.059015+0000","last_change":"2024-07-27T14:42:53.137283+0000","last_active":"2024-07-27T15:17:46.059015+0000","last_peered":"2024-07-27T15:17:46.059015+0000","last_clean":"2024-07-27T15:17:46.059015+0000","last_became_active":"2024-07-27T14:42:53.131434+0000","last_became_peered":"2024-07-27T14:42:53.131434+0000","last_unstale":"2024-07-27T15:17:46.059015+0000","last_undegraded":"2024-07-27T15:17:46.059015+0000","last_fullsized":"2024-07-27T15:17:46.059015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:47.643598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.045645+0000","last_change":"2024-07-27T14:42:32.763809+0000","last_active":"2024-07-27T15:17:46.045645+0000","last_peered":"2024-07-27T15:17:46.045645+0000","last_clean":"2024-07-27T15:17:46.045645+0000","last_became_active":"2024-07-27T14:42:32.755122+0000","last_became_peered":"2024-07-27T14:42:32.755122+0000","last_unstale":"2024-07-27T15:17:46.045645+0000","last_undegraded":"2024-07-27T15:17:46.045645+0000","last_fullsized":"2024-07-27T15:17:46.045645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:18:44.350815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2805,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.497222+0000","last_change":"2024-07-27T14:42:34.656806+0000","last_active":"2024-07-27T15:17:48.497222+0000","last_peered":"2024-07-27T15:17:48.497222+0000","last_clean":"2024-07-27T15:17:48.497222+0000","last_became_active":"2024-07-27T14:42:34.647534+0000","last_became_peered":"2024-07-27T14:42:34.647534+0000","last_unstale":"2024-07-27T15:17:48.497222+0000","last_undegraded":"2024-07-27T15:17:48.497222+0000","last_fullsized":"2024-07-27T15:17:48.497222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:43:30.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.774307+0000","last_change":"2024-07-27T14:42:37.982514+0000","last_active":"2024-07-27T15:17:47.774307+0000","last_peered":"2024-07-27T15:17:47.774307+0000","last_clean":"2024-07-27T15:17:47.774307+0000","last_became_active":"2024-07-27T14:42:37.973009+0000","last_became_peered":"2024-07-27T14:42:37.973009+0000","last_unstale":"2024-07-27T15:17:47.774307+0000","last_undegraded":"2024-07-27T15:17:47.774307+0000","last_fullsized":"2024-07-27T15:17:47.774307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:18.889623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2818,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.061508+0000","last_change":"2024-07-27T14:42:32.206159+0000","last_active":"2024-07-27T15:17:46.061508+0000","last_peered":"2024-07-27T15:17:46.061508+0000","last_clean":"2024-07-27T15:17:46.061508+0000","last_became_active":"2024-07-27T14:42:32.194637+0000","last_became_peered":"2024-07-27T14:42:32.194637+0000","last_unstale":"2024-07-27T15:17:46.061508+0000","last_undegraded":"2024-07-27T15:17:46.061508+0000","last_fullsized":"2024-07-27T15:17:46.061508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:04:59.701883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.513378+0000","last_change":"2024-07-27T14:42:30.522614+0000","last_active":"2024-07-27T15:17:48.513378+0000","last_peered":"2024-07-27T15:17:48.513378+0000","last_clean":"2024-07-27T15:17:48.513378+0000","last_became_active":"2024-07-27T14:42:30.517011+0000","last_became_peered":"2024-07-27T14:42:30.517011+0000","last_unstale":"2024-07-27T15:17:48.513378+0000","last_undegraded":"2024-07-27T15:17:48.513378+0000","last_fullsized":"2024-07-27T15:17:48.513378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:36:37.080542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2815,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.040712+0000","last_change":"2024-07-27T14:42:44.953045+0000","last_active":"2024-07-27T15:17:46.040712+0000","last_peered":"2024-07-27T15:17:46.040712+0000","last_clean":"2024-07-27T15:17:46.040712+0000","last_became_active":"2024-07-27T14:42:44.426410+0000","last_became_peered":"2024-07-27T14:42:44.426410+0000","last_unstale":"2024-07-27T15:17:46.040712+0000","last_undegraded":"2024-07-27T15:17:46.040712+0000","last_fullsized":"2024-07-27T15:17:46.040712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:38:49.627031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.356862+0000","last_change":"2024-07-27T14:43:09.714646+0000","last_active":"2024-07-27T15:17:49.356862+0000","last_peered":"2024-07-27T15:17:49.356862+0000","last_clean":"2024-07-27T15:17:49.356862+0000","last_became_active":"2024-07-27T14:43:09.707672+0000","last_became_peered":"2024-07-27T14:43:09.707672+0000","last_unstale":"2024-07-27T15:17:49.356862+0000","last_undegraded":"2024-07-27T15:17:49.356862+0000","last_fullsized":"2024-07-27T15:17:49.356862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:09:21.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.367195+0000","last_change":"2024-07-27T14:42:32.528826+0000","last_active":"2024-07-27T15:17:49.367195+0000","last_peered":"2024-07-27T15:17:49.367195+0000","last_clean":"2024-07-27T15:17:49.367195+0000","last_became_active":"2024-07-27T14:42:32.384600+0000","last_became_peered":"2024-07-27T14:42:32.384600+0000","last_unstale":"2024-07-27T15:17:49.367195+0000","last_undegraded":"2024-07-27T15:17:49.367195+0000","last_fullsized":"2024-07-27T15:17:49.367195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:54:24.285647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.502171+0000","last_change":"2024-07-27T14:42:33.230973+0000","last_active":"2024-07-27T15:17:48.502171+0000","last_peered":"2024-07-27T15:17:48.502171+0000","last_clean":"2024-07-27T15:17:48.502171+0000","last_became_active":"2024-07-27T14:42:33.224999+0000","last_became_peered":"2024-07-27T14:42:33.224999+0000","last_unstale":"2024-07-27T15:17:48.502171+0000","last_undegraded":"2024-07-27T15:17:48.502171+0000","last_fullsized":"2024-07-27T15:17:48.502171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:37:06.846510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.349035+0000","last_change":"2024-07-27T14:42:32.891119+0000","last_active":"2024-07-27T15:17:49.349035+0000","last_peered":"2024-07-27T15:17:49.349035+0000","last_clean":"2024-07-27T15:17:49.349035+0000","last_became_active":"2024-07-27T14:42:32.853532+0000","last_became_peered":"2024-07-27T14:42:32.853532+0000","last_unstale":"2024-07-27T15:17:49.349035+0000","last_undegraded":"2024-07-27T15:17:49.349035+0000","last_fullsized":"2024-07-27T15:17:49.349035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:38:00.975708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":5570560,"data_stored":2637707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5468,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5599232,"data_stored":2652092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509876,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72720,"kb_used_data":5444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299120,"statfs":{"total":96636764160,"available":96562298880,"internally_reserved":0,"allocated":5574656,"data_stored":2637712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73136,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298704,"statfs":{"total":96636764160,"available":96561872896,"internally_reserved":0,"allocated":5996544,"data_stored":3049927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76904,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294936,"statfs":{"total":96636764160,"available":96558014464,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575282,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":5832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":5971968,"data_stored":3039624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575285,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72708,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299132,"statfs":{"total":96636764160,"available":96562311168,"internally_reserved":0,"allocated":5554176,"data_stored":2630236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T15:17:51.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T15:17:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.141+0000 7ff650d35640 10 monclient: get_monmap_and_config 2024-07-27T15:17:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.141+0000 7ff650d35640 10 monclient: build_initial_monmap 2024-07-27T15:17:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.141+0000 7ff650d35640 10 monclient: monmap: 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:51.142483+0000 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:51.142483+0000 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:51.143 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 1 Processor -- start 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 1 -- start start 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 10 monclient: init 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.142+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff64c0b3630 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff64c0aab60 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ff64c0b46c0 0x7ff64c14e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c0b46c0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64c0aa8a0 con 0x7ff64c0aab60 2024-07-27T15:17:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64c143940 con 0x7ff64c0b3630 2024-07-27T15:17:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7ff64c0b3a10 con 0x7ff64c0b46c0 2024-07-27T15:17:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.143+0000 7ff650d35640 10 monclient(hunting): authenticate will time out at 3411.169094s 2024-07-27T15:17:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.148+0000 7ff643fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7ff64c0aab60 0x7ff64c13ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:39382/0 (socket says 172.21.15.87:39382) 2024-07-27T15:17:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.148+0000 7ff643fff640 1 -- 172.21.15.87:0/3015866453 learned_addr learned my addr 172.21.15.87:0/3015866453 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:51.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.149+0000 7ff64ad9d640 1 --2- 172.21.15.87:0/3015866453 >> v2:172.21.15.87:3300/0 conn(0x7ff64c0b46c0 0x7ff64c14e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.150+0000 7ff64ad9d640 10 monclient(hunting): get_auth_request con 0x7ff64c0b46c0 auth_method 0 2024-07-27T15:17:51.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.150+0000 7ff64ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:51.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.150+0000 7ff64ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:51.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.150+0000 7ff64ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:51.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.151+0000 7ff64ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:51.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.151+0000 7ff64ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:51.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.151+0000 7ff64ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:51.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 10 monclient(hunting): handle_auth_done global_id 14730 payload 274 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 1 -- 172.21.15.87:0/3015866453 >> v1:172.21.15.112:6789/0 conn(0x7ff64c0b3630 legacy=0x7ff64c143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 1 -- 172.21.15.87:0/3015866453 >> v1:172.21.15.87:6789/0 conn(0x7ff64c0aab60 legacy=0x7ff64c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 10 monclient: _finish_hunting 0 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 1 monclient: found mon.noname-a 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:51.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.160+0000 7ff64ad9d640 1 -- 172.21.15.87:0/3015866453 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c140620 con 0x7ff64c0b46c0 2024-07-27T15:17:51.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.167+0000 7ff64ad9d640 1 --2- 172.21.15.87:0/3015866453 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b46c0 0x7ff64c14e420 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d90 tx=0x7ff64402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5ef850e086a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 1 -- 172.21.15.87:0/3015866453 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644039070 con 0x7ff64c0b46c0 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: dump: 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.178 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: _finish_auth 0 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: _check_auth_tickets 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 1 -- 172.21.15.87:0/3015866453 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64402cc60 con 0x7ff64c0b46c0 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 5 monclient: authenticate success, global_id 14730 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 4 set_mon_vals no callback set 2024-07-27T15:17:51.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 1 -- 172.21.15.87:0/3015866453 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644034660 con 0x7ff64c0b46c0 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff649d9b640 10 monclient: dump: 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.180 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.181 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.181 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 10 monclient: shutdown 2024-07-27T15:17:51.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:51.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 -- 172.21.15.87:0/3015866453 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b46c0 msgr2=0x7ff64c14e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 --2- 172.21.15.87:0/3015866453 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b46c0 0x7ff64c14e420 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d90 tx=0x7ff64402beb0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 -- 172.21.15.87:0/3015866453 shutdown_connections 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 --2- 172.21.15.87:0/3015866453 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b46c0 0x7ff64c14e420 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 -- 172.21.15.87:0/3015866453 >> 172.21.15.87:0/3015866453 conn(0x7ff64c0730a0 msgr2=0x7ff64c067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 -- 172.21.15.87:0/3015866453 shutdown_connections 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.177+0000 7ff650d35640 1 -- 172.21.15.87:0/3015866453 wait complete. 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.178+0000 7ff650d35640 10 monclient: build_initial_monmap 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.178+0000 7ff650d35640 10 monclient: monmap: 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.178+0000 7ff650d35640 1 librados: starting msgr at 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.178+0000 7ff650d35640 1 librados: starting objecter 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 Processor -- start 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 -- start start 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 librados: setting wanted keys 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 librados: calling monclient init 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient: init 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff64c0aab60 0x7ff64c1d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c0aab60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c0b3630 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff64c0b46c0 0x7ff64c0b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c0b46c0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c14e960 con 0x7ff64c0b3630 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c142da0 con 0x7ff64c0b46c0 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c14eae0 con 0x7ff64c0aab60 2024-07-27T15:17:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.179+0000 7ff650d35640 10 monclient(hunting): authenticate will time out at 3411.205774s 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.184+0000 7ff643fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37386/0 (socket says 172.21.15.87:37386) 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 1 -- 172.21.15.87:0/1394664085 learned_addr learned my addr 172.21.15.87:0/1394664085 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 10 monclient(hunting): get_auth_request con 0x7ff64c0b3630 auth_method 0 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.186+0000 7ff643fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:51.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.187+0000 7ff643fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:51.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.187+0000 7ff643fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:51.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.187+0000 7ff643fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.189+0000 7ff64a59c640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff64c0aab60 0x7ff64c1d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 10 monclient(hunting): handle_auth_done global_id 14736 payload 824 2024-07-27T15:17:51.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 -- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff64c0aab60 msgr2=0x7ff64c1d85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff64c0aab60 0x7ff64c1d85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 -- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff64c0b46c0 msgr2=0x7ff64c0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff64c0b46c0 0x7ff64c0b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 10 monclient: _finish_hunting 0 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 monclient: found mon.noname-a 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.192+0000 7ff643fff640 1 -- 172.21.15.87:0/1394664085 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c0af830 con 0x7ff64c0b3630 2024-07-27T15:17:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.199+0000 7ff643fff640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c430 tx=0x7ff63400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:51.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.199+0000 7ff642ffd640 10 client.?.objecter ms_handle_connect 0x7ff64c0b3630 2024-07-27T15:17:51.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.199+0000 7ff642ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634017070 con 0x7ff64c0b3630 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: dump: 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff634005ce0 con 0x7ff64c0b3630 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 5 monclient: authenticate success, global_id 14736 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 10 monclient: _renew_subs 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c0ad3e0 con 0x7ff64c0b3630 2024-07-27T15:17:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6340072f0 con 0x7ff64c0b3630 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff642ffd640 10 monclient: dump: 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.213 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff64959a640 4 set_mon_vals no callback set 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 10 client.14736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 10 monclient: _renew_subs 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c1d8cd0 con 0x7ff64c0b3630 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.209+0000 7ff650d35640 1 librados: init done 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.210+0000 7ff650d35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.210+0000 7ff650d35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.210+0000 7ff650d35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.210+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64c13ff10 con 0x7ff64c0b3630 2024-07-27T15:17:51.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.474+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff63401a070 con 0x7ff64c0b3630 2024-07-27T15:17:51.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.474+0000 7ff642ffd640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 0x7ff62c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.475+0000 7ff64a59c640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 0x7ff62c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.475+0000 7ff64a59c640 10 monclient: get_auth_request con 0x7ff62c06d810 auth_method 0 2024-07-27T15:17:51.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.475+0000 7ff64a59c640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 0x7ff62c06fcd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000f60 tx=0x7ff63c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:51.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.475+0000 7ff642ffd640 10 client.14736.objecter ms_handle_connect 0x7ff62c06d810 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7ff634056990 con 0x7ff64c0b3630 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 10 client.14736.objecter ms_dispatch 0x7ff64c000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 3 client.14736.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 3 client.14736.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 20 client.14736.objecter dump_active .. 0 homeless 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7ff634021020 con 0x7ff64c0b3630 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:51.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.489+0000 7ff642ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:51.663 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:54.529 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:51.666 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:54.532 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:51.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.704+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff64c1d7ef0 con 0x7ff62c06d810 2024-07-27T15:17:51.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.708+0000 7ff642ffd640 1 -- 172.21.15.87:0/1394664085 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116367 (secure 0 0 0) 0x7ff64c1d7ef0 con 0x7ff62c06d810 2024-07-27T15:17:51.710 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:17:51.710 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T15:17:51.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 10 librados: watch_flush enter 2024-07-27T15:17:51.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 10 librados: watch_flush exit 2024-07-27T15:17:51.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 20 client.14736.objecter shutdown clearing up homeless session... 2024-07-27T15:17:51.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 10 client.14736.objecter successfully canceled tick 2024-07-27T15:17:51.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 msgr2=0x7ff62c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 0x7ff62c06fcd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000f60 tx=0x7ff63c009040 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 10 monclient: shutdown 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 msgr2=0x7ff64c0b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c430 tx=0x7ff63400c8f0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 shutdown_connections 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff62c06d810 0x7ff62c06fcd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff64c0b46c0 0x7ff64c0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64c0b3630 0x7ff64c0b0420 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 --2- 172.21.15.87:0/1394664085 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff64c0aab60 0x7ff64c1d85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 >> 172.21.15.87:0/1394664085 conn(0x7ff64c0730a0 msgr2=0x7ff64c0b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:51.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 shutdown_connections 2024-07-27T15:17:51.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 -- 172.21.15.87:0/1394664085 wait complete. 2024-07-27T15:17:51.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.710+0000 7ff650d35640 1 librados: shutdown 2024-07-27T15:17:51.723 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":2577,"stamp":"2024-07-27T15:17:50.493597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":122,"num_write_kb":1381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595096,"kb_used_data":44392,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754379624,"statfs":{"total":773094113280,"available":772484734976,"internally_reserved":0,"allocated":45457408,"data_stored":21951472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563793456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":182,"apply_latency_ms":182,"commit_latency_ns":182000000,"apply_latency_ns":182000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.563109"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":2814,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.707879+0000","last_change":"2024-07-27T14:42:41.580179+0000","last_active":"2024-07-27T15:17:47.707879+0000","last_peered":"2024-07-27T15:17:47.707879+0000","last_clean":"2024-07-27T15:17:47.707879+0000","last_became_active":"2024-07-27T14:42:41.574286+0000","last_became_peered":"2024-07-27T14:42:41.574286+0000","last_unstale":"2024-07-27T15:17:47.707879+0000","last_undegraded":"2024-07-27T15:17:47.707879+0000","last_fullsized":"2024-07-27T15:17:47.707879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:03:00.545391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":2822,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.768072+0000","last_change":"2024-07-27T14:42:33.146971+0000","last_active":"2024-07-27T15:17:47.768072+0000","last_peered":"2024-07-27T15:17:47.768072+0000","last_clean":"2024-07-27T15:17:47.768072+0000","last_became_active":"2024-07-27T14:42:33.130123+0000","last_became_peered":"2024-07-27T14:42:33.130123+0000","last_unstale":"2024-07-27T15:17:47.768072+0000","last_undegraded":"2024-07-27T15:17:47.768072+0000","last_fullsized":"2024-07-27T15:17:47.768072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:20:51.941325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2826,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.956847+0000","last_change":"2024-07-27T14:42:32.205230+0000","last_active":"2024-07-27T15:17:45.956847+0000","last_peered":"2024-07-27T15:17:45.956847+0000","last_clean":"2024-07-27T15:17:45.956847+0000","last_became_active":"2024-07-27T14:42:32.196113+0000","last_became_peered":"2024-07-27T14:42:32.196113+0000","last_unstale":"2024-07-27T15:17:45.956847+0000","last_undegraded":"2024-07-27T15:17:45.956847+0000","last_fullsized":"2024-07-27T15:17:45.956847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:11:27.161409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2825,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.975445+0000","last_change":"2024-07-27T14:42:33.556536+0000","last_active":"2024-07-27T15:17:45.975445+0000","last_peered":"2024-07-27T15:17:45.975445+0000","last_clean":"2024-07-27T15:17:45.975445+0000","last_became_active":"2024-07-27T14:42:33.550008+0000","last_became_peered":"2024-07-27T14:42:33.550008+0000","last_unstale":"2024-07-27T15:17:45.975445+0000","last_undegraded":"2024-07-27T15:17:45.975445+0000","last_fullsized":"2024-07-27T15:17:45.975445+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:19:45.505942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":2827,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.697983+0000","last_change":"2024-07-27T14:42:33.671344+0000","last_active":"2024-07-27T15:17:47.697983+0000","last_peered":"2024-07-27T15:17:47.697983+0000","last_clean":"2024-07-27T15:17:47.697983+0000","last_became_active":"2024-07-27T14:42:33.554937+0000","last_became_peered":"2024-07-27T14:42:33.554937+0000","last_unstale":"2024-07-27T15:17:47.697983+0000","last_undegraded":"2024-07-27T15:17:47.697983+0000","last_fullsized":"2024-07-27T15:17:47.697983+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:25:20.740493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2802,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.100193+0000","last_change":"2024-07-27T14:42:43.884287+0000","last_active":"2024-07-27T15:17:48.100193+0000","last_peered":"2024-07-27T15:17:48.100193+0000","last_clean":"2024-07-27T15:17:48.100193+0000","last_became_active":"2024-07-27T14:42:43.877378+0000","last_became_peered":"2024-07-27T14:42:43.877378+0000","last_unstale":"2024-07-27T15:17:48.100193+0000","last_undegraded":"2024-07-27T15:17:48.100193+0000","last_fullsized":"2024-07-27T15:17:48.100193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:37:47.388561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":2817,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.369035+0000","last_change":"2024-07-27T14:42:35.229319+0000","last_active":"2024-07-27T15:17:49.369035+0000","last_peered":"2024-07-27T15:17:49.369035+0000","last_clean":"2024-07-27T15:17:49.369035+0000","last_became_active":"2024-07-27T14:42:35.220195+0000","last_became_peered":"2024-07-27T14:42:35.220195+0000","last_unstale":"2024-07-27T15:17:49.369035+0000","last_undegraded":"2024-07-27T15:17:49.369035+0000","last_fullsized":"2024-07-27T15:17:49.369035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:46:10.403748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.946537+0000","last_change":"2024-07-27T14:42:33.873297+0000","last_active":"2024-07-27T15:17:45.946537+0000","last_peered":"2024-07-27T15:17:45.946537+0000","last_clean":"2024-07-27T15:17:45.946537+0000","last_became_active":"2024-07-27T14:42:33.867602+0000","last_became_peered":"2024-07-27T14:42:33.867602+0000","last_unstale":"2024-07-27T15:17:45.946537+0000","last_undegraded":"2024-07-27T15:17:45.946537+0000","last_fullsized":"2024-07-27T15:17:45.946537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:17:09.550687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.488131+0000","last_change":"2024-07-27T14:42:34.920701+0000","last_active":"2024-07-27T15:17:48.488131+0000","last_peered":"2024-07-27T15:17:48.488131+0000","last_clean":"2024-07-27T15:17:48.488131+0000","last_became_active":"2024-07-27T14:42:34.746905+0000","last_became_peered":"2024-07-27T14:42:34.746905+0000","last_unstale":"2024-07-27T15:17:48.488131+0000","last_undegraded":"2024-07-27T15:17:48.488131+0000","last_fullsized":"2024-07-27T15:17:48.488131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:30:00.579219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.766173+0000","last_change":"2024-07-27T14:42:37.296338+0000","last_active":"2024-07-27T15:17:47.766173+0000","last_peered":"2024-07-27T15:17:47.766173+0000","last_clean":"2024-07-27T15:17:47.766173+0000","last_became_active":"2024-07-27T14:42:37.287685+0000","last_became_peered":"2024-07-27T14:42:37.287685+0000","last_unstale":"2024-07-27T15:17:47.766173+0000","last_undegraded":"2024-07-27T15:17:47.766173+0000","last_fullsized":"2024-07-27T15:17:47.766173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:04:06.389200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":2814,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.360888+0000","last_change":"2024-07-27T14:42:35.360457+0000","last_active":"2024-07-27T15:17:49.360888+0000","last_peered":"2024-07-27T15:17:49.360888+0000","last_clean":"2024-07-27T15:17:49.360888+0000","last_became_active":"2024-07-27T14:42:35.354552+0000","last_became_peered":"2024-07-27T14:42:35.354552+0000","last_unstale":"2024-07-27T15:17:49.360888+0000","last_undegraded":"2024-07-27T15:17:49.360888+0000","last_fullsized":"2024-07-27T15:17:49.360888+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:22:00.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.523171+0000","last_change":"2024-07-27T14:42:34.577783+0000","last_active":"2024-07-27T15:17:48.523171+0000","last_peered":"2024-07-27T15:17:48.523171+0000","last_clean":"2024-07-27T15:17:48.523171+0000","last_became_active":"2024-07-27T14:42:34.568719+0000","last_became_peered":"2024-07-27T14:42:34.568719+0000","last_unstale":"2024-07-27T15:17:48.523171+0000","last_undegraded":"2024-07-27T15:17:48.523171+0000","last_fullsized":"2024-07-27T15:17:48.523171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:14:22.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2929,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.109246+0000","last_change":"2024-07-27T14:41:03.617220+0000","last_active":"2024-07-27T15:17:48.109246+0000","last_peered":"2024-07-27T15:17:48.109246+0000","last_clean":"2024-07-27T15:17:48.109246+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T15:17:48.109246+0000","last_undegraded":"2024-07-27T15:17:48.109246+0000","last_fullsized":"2024-07-27T15:17:48.109246+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_clean_scrub_stamp":"2024-07-27T14:41:00.649818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:14:15.632432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":2847,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.738254+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T15:17:47.738254+0000","last_peered":"2024-07-27T15:17:47.738254+0000","last_clean":"2024-07-27T15:17:47.738254+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T15:17:47.738254+0000","last_undegraded":"2024-07-27T15:17:47.738254+0000","last_fullsized":"2024-07-27T15:17:47.738254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:40:29.539136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":2825,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.769897+0000","last_change":"2024-07-27T14:42:37.311409+0000","last_active":"2024-07-27T15:17:47.769897+0000","last_peered":"2024-07-27T15:17:47.769897+0000","last_clean":"2024-07-27T15:17:47.769897+0000","last_became_active":"2024-07-27T14:42:37.189428+0000","last_became_peered":"2024-07-27T14:42:37.189428+0000","last_unstale":"2024-07-27T15:17:47.769897+0000","last_undegraded":"2024-07-27T15:17:47.769897+0000","last_fullsized":"2024-07-27T15:17:47.769897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:15:12.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2846,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.695050+0000","last_change":"2024-07-27T14:41:21.948784+0000","last_active":"2024-07-27T15:17:47.695050+0000","last_peered":"2024-07-27T15:17:47.695050+0000","last_clean":"2024-07-27T15:17:47.695050+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T15:17:47.695050+0000","last_undegraded":"2024-07-27T15:17:47.695050+0000","last_fullsized":"2024-07-27T15:17:47.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:23:48.411987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.959864+0000","last_change":"2024-07-27T14:42:32.948619+0000","last_active":"2024-07-27T15:17:45.959864+0000","last_peered":"2024-07-27T15:17:45.959864+0000","last_clean":"2024-07-27T15:17:45.959864+0000","last_became_active":"2024-07-27T14:42:32.825779+0000","last_became_peered":"2024-07-27T14:42:32.825779+0000","last_unstale":"2024-07-27T15:17:45.959864+0000","last_undegraded":"2024-07-27T15:17:45.959864+0000","last_fullsized":"2024-07-27T15:17:45.959864+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:44:30.813522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.506184+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T15:17:48.506184+0000","last_peered":"2024-07-27T15:17:48.506184+0000","last_clean":"2024-07-27T15:17:48.506184+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T15:17:48.506184+0000","last_undegraded":"2024-07-27T15:17:48.506184+0000","last_fullsized":"2024-07-27T15:17:48.506184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2813,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.720308+0000","last_change":"2024-07-27T14:43:03.368313+0000","last_active":"2024-07-27T15:17:47.720308+0000","last_peered":"2024-07-27T15:17:47.720308+0000","last_clean":"2024-07-27T15:17:47.720308+0000","last_became_active":"2024-07-27T14:43:03.360033+0000","last_became_peered":"2024-07-27T14:43:03.360033+0000","last_unstale":"2024-07-27T15:17:47.720308+0000","last_undegraded":"2024-07-27T15:17:47.720308+0000","last_fullsized":"2024-07-27T15:17:47.720308+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:21:29.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":2848,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.678919+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T15:17:47.678919+0000","last_peered":"2024-07-27T15:17:47.678919+0000","last_clean":"2024-07-27T15:17:47.678919+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T15:17:47.678919+0000","last_undegraded":"2024-07-27T15:17:47.678919+0000","last_fullsized":"2024-07-27T15:17:47.678919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:39:58.978046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.704667+0000","last_change":"2024-07-27T14:42:33.420873+0000","last_active":"2024-07-27T15:17:47.704667+0000","last_peered":"2024-07-27T15:17:47.704667+0000","last_clean":"2024-07-27T15:17:47.704667+0000","last_became_active":"2024-07-27T14:42:33.415149+0000","last_became_peered":"2024-07-27T14:42:33.415149+0000","last_unstale":"2024-07-27T15:17:47.704667+0000","last_undegraded":"2024-07-27T15:17:47.704667+0000","last_fullsized":"2024-07-27T15:17:47.704667+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:45:18.005668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2848,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.951360+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T15:17:45.951360+0000","last_peered":"2024-07-27T15:17:45.951360+0000","last_clean":"2024-07-27T15:17:45.951360+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T15:17:45.951360+0000","last_undegraded":"2024-07-27T15:17:45.951360+0000","last_fullsized":"2024-07-27T15:17:45.951360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.698776+0000","last_change":"2024-07-27T14:42:57.118705+0000","last_active":"2024-07-27T15:17:47.698776+0000","last_peered":"2024-07-27T15:17:47.698776+0000","last_clean":"2024-07-27T15:17:47.698776+0000","last_became_active":"2024-07-27T14:42:57.112921+0000","last_became_peered":"2024-07-27T14:42:57.112921+0000","last_unstale":"2024-07-27T15:17:47.698776+0000","last_undegraded":"2024-07-27T15:17:47.698776+0000","last_fullsized":"2024-07-27T15:17:47.698776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:02:08.809341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2843,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.771806+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T15:17:47.771806+0000","last_peered":"2024-07-27T15:17:47.771806+0000","last_clean":"2024-07-27T15:17:47.771806+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T15:17:47.771806+0000","last_undegraded":"2024-07-27T15:17:47.771806+0000","last_fullsized":"2024-07-27T15:17:47.771806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:57:17.520382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.702273+0000","last_change":"2024-07-27T14:42:36.978696+0000","last_active":"2024-07-27T15:17:47.702273+0000","last_peered":"2024-07-27T15:17:47.702273+0000","last_clean":"2024-07-27T15:17:47.702273+0000","last_became_active":"2024-07-27T14:42:36.941428+0000","last_became_peered":"2024-07-27T14:42:36.941428+0000","last_unstale":"2024-07-27T15:17:47.702273+0000","last_undegraded":"2024-07-27T15:17:47.702273+0000","last_fullsized":"2024-07-27T15:17:47.702273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:34:55.684245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2835,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.063984+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T15:17:46.063984+0000","last_peered":"2024-07-27T15:17:46.063984+0000","last_clean":"2024-07-27T15:17:46.063984+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T15:17:46.063984+0000","last_undegraded":"2024-07-27T15:17:46.063984+0000","last_fullsized":"2024-07-27T15:17:46.063984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2819,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.010815+0000","last_change":"2024-07-27T14:42:32.399654+0000","last_active":"2024-07-27T15:17:46.010815+0000","last_peered":"2024-07-27T15:17:46.010815+0000","last_clean":"2024-07-27T15:17:46.010815+0000","last_became_active":"2024-07-27T14:42:32.329977+0000","last_became_peered":"2024-07-27T14:42:32.329977+0000","last_unstale":"2024-07-27T15:17:46.010815+0000","last_undegraded":"2024-07-27T15:17:46.010815+0000","last_fullsized":"2024-07-27T15:17:46.010815+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:17:59.414451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2840,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.777392+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T15:17:47.777392+0000","last_peered":"2024-07-27T15:17:47.777392+0000","last_clean":"2024-07-27T15:17:47.777392+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T15:17:47.777392+0000","last_undegraded":"2024-07-27T15:17:47.777392+0000","last_fullsized":"2024-07-27T15:17:47.777392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:05:24.194010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"44'7","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.354492+0000","last_change":"2024-07-27T14:42:34.811194+0000","last_active":"2024-07-27T15:17:49.354492+0000","last_peered":"2024-07-27T15:17:49.354492+0000","last_clean":"2024-07-27T15:17:49.354492+0000","last_became_active":"2024-07-27T14:42:34.791352+0000","last_became_peered":"2024-07-27T14:42:34.791352+0000","last_unstale":"2024-07-27T15:17:49.354492+0000","last_undegraded":"2024-07-27T15:17:49.354492+0000","last_fullsized":"2024-07-27T15:17:49.354492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:36:47.747804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":2816,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.358672+0000","last_change":"2024-07-27T14:42:35.255013+0000","last_active":"2024-07-27T15:17:49.358672+0000","last_peered":"2024-07-27T15:17:49.358672+0000","last_clean":"2024-07-27T15:17:49.358672+0000","last_became_active":"2024-07-27T14:42:35.248761+0000","last_became_peered":"2024-07-27T14:42:35.248761+0000","last_unstale":"2024-07-27T15:17:49.358672+0000","last_undegraded":"2024-07-27T15:17:49.358672+0000","last_fullsized":"2024-07-27T15:17:49.358672+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:43:59.817053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.059015+0000","last_change":"2024-07-27T14:42:53.137283+0000","last_active":"2024-07-27T15:17:46.059015+0000","last_peered":"2024-07-27T15:17:46.059015+0000","last_clean":"2024-07-27T15:17:46.059015+0000","last_became_active":"2024-07-27T14:42:53.131434+0000","last_became_peered":"2024-07-27T14:42:53.131434+0000","last_unstale":"2024-07-27T15:17:46.059015+0000","last_undegraded":"2024-07-27T15:17:46.059015+0000","last_fullsized":"2024-07-27T15:17:46.059015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:47.643598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.045645+0000","last_change":"2024-07-27T14:42:32.763809+0000","last_active":"2024-07-27T15:17:46.045645+0000","last_peered":"2024-07-27T15:17:46.045645+0000","last_clean":"2024-07-27T15:17:46.045645+0000","last_became_active":"2024-07-27T14:42:32.755122+0000","last_became_peered":"2024-07-27T14:42:32.755122+0000","last_unstale":"2024-07-27T15:17:46.045645+0000","last_undegraded":"2024-07-27T15:17:46.045645+0000","last_fullsized":"2024-07-27T15:17:46.045645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:18:44.350815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2805,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.497222+0000","last_change":"2024-07-27T14:42:34.656806+0000","last_active":"2024-07-27T15:17:48.497222+0000","last_peered":"2024-07-27T15:17:48.497222+0000","last_clean":"2024-07-27T15:17:48.497222+0000","last_became_active":"2024-07-27T14:42:34.647534+0000","last_became_peered":"2024-07-27T14:42:34.647534+0000","last_unstale":"2024-07-27T15:17:48.497222+0000","last_undegraded":"2024-07-27T15:17:48.497222+0000","last_fullsized":"2024-07-27T15:17:48.497222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:43:30.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.774307+0000","last_change":"2024-07-27T14:42:37.982514+0000","last_active":"2024-07-27T15:17:47.774307+0000","last_peered":"2024-07-27T15:17:47.774307+0000","last_clean":"2024-07-27T15:17:47.774307+0000","last_became_active":"2024-07-27T14:42:37.973009+0000","last_became_peered":"2024-07-27T14:42:37.973009+0000","last_unstale":"2024-07-27T15:17:47.774307+0000","last_undegraded":"2024-07-27T15:17:47.774307+0000","last_fullsized":"2024-07-27T15:17:47.774307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:18.889623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2818,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.061508+0000","last_change":"2024-07-27T14:42:32.206159+0000","last_active":"2024-07-27T15:17:46.061508+0000","last_peered":"2024-07-27T15:17:46.061508+0000","last_clean":"2024-07-27T15:17:46.061508+0000","last_became_active":"2024-07-27T14:42:32.194637+0000","last_became_peered":"2024-07-27T14:42:32.194637+0000","last_unstale":"2024-07-27T15:17:46.061508+0000","last_undegraded":"2024-07-27T15:17:46.061508+0000","last_fullsized":"2024-07-27T15:17:46.061508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:04:59.701883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.513378+0000","last_change":"2024-07-27T14:42:30.522614+0000","last_active":"2024-07-27T15:17:48.513378+0000","last_peered":"2024-07-27T15:17:48.513378+0000","last_clean":"2024-07-27T15:17:48.513378+0000","last_became_active":"2024-07-27T14:42:30.517011+0000","last_became_peered":"2024-07-27T14:42:30.517011+0000","last_unstale":"2024-07-27T15:17:48.513378+0000","last_undegraded":"2024-07-27T15:17:48.513378+0000","last_fullsized":"2024-07-27T15:17:48.513378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:36:37.080542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2815,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.040712+0000","last_change":"2024-07-27T14:42:44.953045+0000","last_active":"2024-07-27T15:17:46.040712+0000","last_peered":"2024-07-27T15:17:46.040712+0000","last_clean":"2024-07-27T15:17:46.040712+0000","last_became_active":"2024-07-27T14:42:44.426410+0000","last_became_peered":"2024-07-27T14:42:44.426410+0000","last_unstale":"2024-07-27T15:17:46.040712+0000","last_undegraded":"2024-07-27T15:17:46.040712+0000","last_fullsized":"2024-07-27T15:17:46.040712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:38:49.627031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.356862+0000","last_change":"2024-07-27T14:43:09.714646+0000","last_active":"2024-07-27T15:17:49.356862+0000","last_peered":"2024-07-27T15:17:49.356862+0000","last_clean":"2024-07-27T15:17:49.356862+0000","last_became_active":"2024-07-27T14:43:09.707672+0000","last_became_peered":"2024-07-27T14:43:09.707672+0000","last_unstale":"2024-07-27T15:17:49.356862+0000","last_undegraded":"2024-07-27T15:17:49.356862+0000","last_fullsized":"2024-07-27T15:17:49.356862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:09:21.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.367195+0000","last_change":"2024-07-27T14:42:32.528826+0000","last_active":"2024-07-27T15:17:49.367195+0000","last_peered":"2024-07-27T15:17:49.367195+0000","last_clean":"2024-07-27T15:17:49.367195+0000","last_became_active":"2024-07-27T14:42:32.384600+0000","last_became_peered":"2024-07-27T14:42:32.384600+0000","last_unstale":"2024-07-27T15:17:49.367195+0000","last_undegraded":"2024-07-27T15:17:49.367195+0000","last_fullsized":"2024-07-27T15:17:49.367195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:54:24.285647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.502171+0000","last_change":"2024-07-27T14:42:33.230973+0000","last_active":"2024-07-27T15:17:48.502171+0000","last_peered":"2024-07-27T15:17:48.502171+0000","last_clean":"2024-07-27T15:17:48.502171+0000","last_became_active":"2024-07-27T14:42:33.224999+0000","last_became_peered":"2024-07-27T14:42:33.224999+0000","last_unstale":"2024-07-27T15:17:48.502171+0000","last_undegraded":"2024-07-27T15:17:48.502171+0000","last_fullsized":"2024-07-27T15:17:48.502171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:37:06.846510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.349035+0000","last_change":"2024-07-27T14:42:32.891119+0000","last_active":"2024-07-27T15:17:49.349035+0000","last_peered":"2024-07-27T15:17:49.349035+0000","last_clean":"2024-07-27T15:17:49.349035+0000","last_became_active":"2024-07-27T14:42:32.853532+0000","last_became_peered":"2024-07-27T14:42:32.853532+0000","last_unstale":"2024-07-27T15:17:49.349035+0000","last_undegraded":"2024-07-27T15:17:49.349035+0000","last_fullsized":"2024-07-27T15:17:49.349035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:38:00.975708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":5570560,"data_stored":2637707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5468,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5599232,"data_stored":2652092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509876,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72720,"kb_used_data":5444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299120,"statfs":{"total":96636764160,"available":96562298880,"internally_reserved":0,"allocated":5574656,"data_stored":2637712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73136,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298704,"statfs":{"total":96636764160,"available":96561872896,"internally_reserved":0,"allocated":5996544,"data_stored":3049927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76904,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294936,"statfs":{"total":96636764160,"available":96558014464,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575282,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":5832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":5971968,"data_stored":3039624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575285,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72708,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299132,"statfs":{"total":96636764160,"available":96562311168,"internally_reserved":0,"allocated":5554176,"data_stored":2630236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T15:17:51.726 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-27T15:17:51.726 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T15:17:51.734 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:37:54.600 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:17:51.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.828+0000 7f7c082cc640 10 monclient: get_monmap_and_config 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.828+0000 7f7c082cc640 10 monclient: build_initial_monmap 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.828+0000 7f7c082cc640 10 monclient: monmap: 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:51.829822+0000 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:51.829822+0000 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:51.830 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:51.831 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:51.831 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.829+0000 7f7c082cc640 1 Processor -- start 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.829+0000 7f7c082cc640 1 -- start start 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.829+0000 7f7c082cc640 10 monclient: init 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-07-27T15:17:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c000b5aa0 0x7f7c000aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-c con 0x7f7c000b5aa0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f7c000b2e30 0x7f7c000b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000b2e30 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c00146190 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c000b61b0 con 0x7f7c00146190 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f7c000b5e80 con 0x7f7c000b2e30 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f7c000b52c0 con 0x7f7c000b5aa0 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c082cc640 10 monclient(hunting): authenticate will time out at 3411.856246s 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.830+0000 7f7c06068640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c000b5aa0 0x7f7c000aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c000b5aa0 0x7f7c000aaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:55724/0 (socket says 172.21.15.87:55724) 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 1 -- 172.21.15.87:0/4050913083 learned_addr learned my addr 172.21.15.87:0/4050913083 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): get_auth_request con 0x7f7c000b5aa0 auth_method 0 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:51.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.831+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:51.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 10 monclient(hunting): handle_auth_done global_id 21880 payload 274 2024-07-27T15:17:51.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 1 -- 172.21.15.87:0/4050913083 >> v2:172.21.15.87:3300/0 conn(0x7f7c000b2e30 msgr2=0x7f7c000b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:51.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 1 --2- 172.21.15.87:0/4050913083 >> v2:172.21.15.87:3300/0 conn(0x7f7c000b2e30 0x7f7c000b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 1 -- 172.21.15.87:0/4050913083 >> v1:172.21.15.87:6789/0 conn(0x7f7c00146190 legacy=0x7f7c0013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:51.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 10 monclient: _finish_hunting 0 2024-07-27T15:17:51.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 1 monclient: found mon.noname-c 2024-07-27T15:17:51.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:51.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:51.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.833+0000 7f7c06068640 1 -- 172.21.15.87:0/4050913083 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c00146620 con 0x7f7c000b5aa0 2024-07-27T15:17:51.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.835+0000 7f7c06068640 1 --2- 172.21.15.87:0/4050913083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b5aa0 0x7f7c000aaa50 secure :-1 s=READY pgs=5015 cs=0 l=1 rev1=1 crypto rx=0x7f7c000b9a20 tx=0x7f7bfc02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e26c608d68bfca25 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 1 -- 172.21.15.87:0/4050913083 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc038070 con 0x7f7c000b5aa0 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: dump: 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.840 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: _finish_auth 0 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: _check_auth_tickets 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 1 -- 172.21.15.87:0/4050913083 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bfc02f050 con 0x7f7c000b5aa0 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 5 monclient: authenticate success, global_id 21880 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 1 -- 172.21.15.87:0/4050913083 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc031970 con 0x7f7c000b5aa0 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 4 set_mon_vals no callback set 2024-07-27T15:17:51.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:51.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c05066640 10 monclient: dump: 2024-07-27T15:17:51.842 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.842 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.842 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.842 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.843 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 10 monclient: shutdown 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 -- 172.21.15.87:0/4050913083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b5aa0 msgr2=0x7f7c000aaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 --2- 172.21.15.87:0/4050913083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b5aa0 0x7f7c000aaa50 secure :-1 s=READY pgs=5015 cs=0 l=1 rev1=1 crypto rx=0x7f7c000b9a20 tx=0x7f7bfc02b7f0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 -- 172.21.15.87:0/4050913083 shutdown_connections 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 --2- 172.21.15.87:0/4050913083 >> v2:172.21.15.87:3300/0 conn(0x7f7c000b2e30 0x7f7c000b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 --2- 172.21.15.87:0/4050913083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b5aa0 0x7f7c000aaa50 unknown :-1 s=CLOSED pgs=5015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 -- 172.21.15.87:0/4050913083 >> 172.21.15.87:0/4050913083 conn(0x7f7c00067810 msgr2=0x7f7c00067c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 -- 172.21.15.87:0/4050913083 shutdown_connections 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.839+0000 7f7c082cc640 1 -- 172.21.15.87:0/4050913083 wait complete. 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 10 monclient: build_initial_monmap 2024-07-27T15:17:51.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 10 monclient: monmap: 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 librados: starting msgr at 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 librados: starting objecter 2024-07-27T15:17:51.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 Processor -- start 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 -- start start 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 librados: setting wanted keys 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 1 librados: calling monclient init 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.840+0000 7f7c082cc640 10 monclient: init 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-b con 0x7f7c000b2e30 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c000b5aa0 0x7f7c001dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000b5aa0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7c00146190 0x7f7c001dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): picked mon.noname-c con 0x7f7c00146190 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:51.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0014d4e0 con 0x7f7c000b5aa0 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c000b48f0 con 0x7f7c000b2e30 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0014d660 con 0x7f7c00146190 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c082cc640 10 monclient(hunting): authenticate will time out at 3411.867380s 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c06869640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7c00146190 0x7f7c001dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.841+0000 7f7c06068640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:55726/0 (socket says 172.21.15.87:55726) 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 1 -- 172.21.15.87:0/3620262021 learned_addr learned my addr 172.21.15.87:0/3620262021 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): get_auth_request con 0x7f7c000b2e30 auth_method 0 2024-07-27T15:17:51.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): get_auth_request con 0x7f7c00146190 auth_method 0 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.842+0000 7f7c06869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 10 monclient(hunting): handle_auth_done global_id 21883 payload 824 2024-07-27T15:17:51.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 -- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7c00146190 msgr2=0x7f7c001dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7c00146190 0x7f7c001dc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 -- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c000b5aa0 msgr2=0x7f7c001dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c000b5aa0 0x7f7c001dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 10 monclient: _finish_hunting 0 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 monclient: found mon.noname-b 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:51.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.845+0000 7f7c06068640 1 -- 172.21.15.87:0/3620262021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c00149000 con 0x7f7c000b2e30 2024-07-27T15:17:51.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.847+0000 7f7c06068640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 secure :-1 s=READY pgs=5016 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc02b830 tx=0x7f7bfc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:51.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.847+0000 7f7bf6ffd640 10 client.?.objecter ms_handle_connect 0x7f7c000b2e30 2024-07-27T15:17:51.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.847+0000 7f7bf6ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc043070 con 0x7f7c000b2e30 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: dump: 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.856 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.858 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.858 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.858 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.858 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bfc03f870 con 0x7f7c000b2e30 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 5 monclient: authenticate success, global_id 21883 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 10 monclient: _renew_subs 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:51.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0014a3b0 con 0x7f7c000b2e30 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc038040 con 0x7f7c000b2e30 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7bf6ffd640 10 monclient: dump: 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:51.860 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c04865640 4 set_mon_vals no callback set 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 10 client.21883.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 10 monclient: _renew_subs 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c001489b0 con 0x7f7c000b2e30 2024-07-27T15:17:51.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.854+0000 7f7c082cc640 1 librados: init done 2024-07-27T15:17:51.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.855+0000 7f7c082cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:51.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.855+0000 7f7c082cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:51.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.855+0000 7f7c082cc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:51.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:51.855+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c000b4d80 con 0x7f7c000b2e30 2024-07-27T15:17:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.103+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7bfc046070 con 0x7f7c000b2e30 2024-07-27T15:17:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.104+0000 7f7bf6ffd640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 0x7f7be406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.104+0000 7f7c05867640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 0x7f7be406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.104+0000 7f7c05867640 10 monclient: get_auth_request con 0x7f7be406d810 auth_method 0 2024-07-27T15:17:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.105+0000 7f7bf6ffd640 10 client.21883.objecter ms_handle_connect 0x7f7be406d810 2024-07-27T15:17:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.105+0000 7f7c05867640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 0x7f7be406fcd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c00143a90 tx=0x7f7bf0006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:52.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.107+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f7bfc079820 con 0x7f7c000b2e30 2024-07-27T15:17:52.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.107+0000 7f7bf6ffd640 10 client.21883.objecter ms_dispatch 0x7f7c00000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:52.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.107+0000 7f7bf6ffd640 3 client.21883.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:52.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.107+0000 7f7bf6ffd640 3 client.21883.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:52.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.107+0000 7f7bf6ffd640 20 client.21883.objecter dump_active .. 0 homeless 2024-07-27T15:17:52.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.117+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f7bfc07b780 con 0x7f7c000b2e30 2024-07-27T15:17:52.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.117+0000 7f7bf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:52.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.117+0000 7f7bf6ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:52.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.301+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c000b84c0 con 0x7f7be406d810 2024-07-27T15:17:52.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.304+0000 7f7bf6ffd640 1 -- 172.21.15.87:0/3620262021 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116367 (secure 0 0 0) 0x7f7c000b84c0 con 0x7f7be406d810 2024-07-27T15:17:52.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:17:52.307 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 10 librados: watch_flush enter 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 10 librados: watch_flush exit 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 20 client.21883.objecter shutdown clearing up homeless session... 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 10 client.21883.objecter successfully canceled tick 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 msgr2=0x7f7be406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 0x7f7be406fcd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c00143a90 tx=0x7f7bf0006c90 comp rx=0 tx=0).stop 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 10 monclient: shutdown 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 msgr2=0x7f7c00148030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 secure :-1 s=READY pgs=5016 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc02b830 tx=0x7f7bfc0046c0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 shutdown_connections 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f7be406d810 0x7f7be406fcd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7c00146190 0x7f7c001dc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c000b5aa0 0x7f7c001dbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 --2- 172.21.15.87:0/3620262021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c000b2e30 0x7f7c00148030 unknown :-1 s=CLOSED pgs=5016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 >> 172.21.15.87:0/3620262021 conn(0x7f7c00067810 msgr2=0x7f7c0014c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 shutdown_connections 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 -- 172.21.15.87:0/3620262021 wait complete. 2024-07-27T15:17:52.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.307+0000 7f7c082cc640 1 librados: shutdown 2024-07-27T15:17:52.319 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":2577,"stamp":"2024-07-27T15:17:50.493597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":122,"num_write_kb":1381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595096,"kb_used_data":44392,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754379624,"statfs":{"total":773094113280,"available":772484734976,"internally_reserved":0,"allocated":45457408,"data_stored":21951472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563793456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":182,"apply_latency_ms":182,"commit_latency_ns":182000000,"apply_latency_ns":182000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.563109"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":2814,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.707879+0000","last_change":"2024-07-27T14:42:41.580179+0000","last_active":"2024-07-27T15:17:47.707879+0000","last_peered":"2024-07-27T15:17:47.707879+0000","last_clean":"2024-07-27T15:17:47.707879+0000","last_became_active":"2024-07-27T14:42:41.574286+0000","last_became_peered":"2024-07-27T14:42:41.574286+0000","last_unstale":"2024-07-27T15:17:47.707879+0000","last_undegraded":"2024-07-27T15:17:47.707879+0000","last_fullsized":"2024-07-27T15:17:47.707879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:03:00.545391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":2822,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.768072+0000","last_change":"2024-07-27T14:42:33.146971+0000","last_active":"2024-07-27T15:17:47.768072+0000","last_peered":"2024-07-27T15:17:47.768072+0000","last_clean":"2024-07-27T15:17:47.768072+0000","last_became_active":"2024-07-27T14:42:33.130123+0000","last_became_peered":"2024-07-27T14:42:33.130123+0000","last_unstale":"2024-07-27T15:17:47.768072+0000","last_undegraded":"2024-07-27T15:17:47.768072+0000","last_fullsized":"2024-07-27T15:17:47.768072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:20:51.941325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2826,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.956847+0000","last_change":"2024-07-27T14:42:32.205230+0000","last_active":"2024-07-27T15:17:45.956847+0000","last_peered":"2024-07-27T15:17:45.956847+0000","last_clean":"2024-07-27T15:17:45.956847+0000","last_became_active":"2024-07-27T14:42:32.196113+0000","last_became_peered":"2024-07-27T14:42:32.196113+0000","last_unstale":"2024-07-27T15:17:45.956847+0000","last_undegraded":"2024-07-27T15:17:45.956847+0000","last_fullsized":"2024-07-27T15:17:45.956847+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:11:27.161409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2825,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.975445+0000","last_change":"2024-07-27T14:42:33.556536+0000","last_active":"2024-07-27T15:17:45.975445+0000","last_peered":"2024-07-27T15:17:45.975445+0000","last_clean":"2024-07-27T15:17:45.975445+0000","last_became_active":"2024-07-27T14:42:33.550008+0000","last_became_peered":"2024-07-27T14:42:33.550008+0000","last_unstale":"2024-07-27T15:17:45.975445+0000","last_undegraded":"2024-07-27T15:17:45.975445+0000","last_fullsized":"2024-07-27T15:17:45.975445+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:19:45.505942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":2827,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.697983+0000","last_change":"2024-07-27T14:42:33.671344+0000","last_active":"2024-07-27T15:17:47.697983+0000","last_peered":"2024-07-27T15:17:47.697983+0000","last_clean":"2024-07-27T15:17:47.697983+0000","last_became_active":"2024-07-27T14:42:33.554937+0000","last_became_peered":"2024-07-27T14:42:33.554937+0000","last_unstale":"2024-07-27T15:17:47.697983+0000","last_undegraded":"2024-07-27T15:17:47.697983+0000","last_fullsized":"2024-07-27T15:17:47.697983+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:25:20.740493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2802,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.100193+0000","last_change":"2024-07-27T14:42:43.884287+0000","last_active":"2024-07-27T15:17:48.100193+0000","last_peered":"2024-07-27T15:17:48.100193+0000","last_clean":"2024-07-27T15:17:48.100193+0000","last_became_active":"2024-07-27T14:42:43.877378+0000","last_became_peered":"2024-07-27T14:42:43.877378+0000","last_unstale":"2024-07-27T15:17:48.100193+0000","last_undegraded":"2024-07-27T15:17:48.100193+0000","last_fullsized":"2024-07-27T15:17:48.100193+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:37:47.388561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":2817,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.369035+0000","last_change":"2024-07-27T14:42:35.229319+0000","last_active":"2024-07-27T15:17:49.369035+0000","last_peered":"2024-07-27T15:17:49.369035+0000","last_clean":"2024-07-27T15:17:49.369035+0000","last_became_active":"2024-07-27T14:42:35.220195+0000","last_became_peered":"2024-07-27T14:42:35.220195+0000","last_unstale":"2024-07-27T15:17:49.369035+0000","last_undegraded":"2024-07-27T15:17:49.369035+0000","last_fullsized":"2024-07-27T15:17:49.369035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:46:10.403748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.946537+0000","last_change":"2024-07-27T14:42:33.873297+0000","last_active":"2024-07-27T15:17:45.946537+0000","last_peered":"2024-07-27T15:17:45.946537+0000","last_clean":"2024-07-27T15:17:45.946537+0000","last_became_active":"2024-07-27T14:42:33.867602+0000","last_became_peered":"2024-07-27T14:42:33.867602+0000","last_unstale":"2024-07-27T15:17:45.946537+0000","last_undegraded":"2024-07-27T15:17:45.946537+0000","last_fullsized":"2024-07-27T15:17:45.946537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:17:09.550687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.488131+0000","last_change":"2024-07-27T14:42:34.920701+0000","last_active":"2024-07-27T15:17:48.488131+0000","last_peered":"2024-07-27T15:17:48.488131+0000","last_clean":"2024-07-27T15:17:48.488131+0000","last_became_active":"2024-07-27T14:42:34.746905+0000","last_became_peered":"2024-07-27T14:42:34.746905+0000","last_unstale":"2024-07-27T15:17:48.488131+0000","last_undegraded":"2024-07-27T15:17:48.488131+0000","last_fullsized":"2024-07-27T15:17:48.488131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:30:00.579219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.766173+0000","last_change":"2024-07-27T14:42:37.296338+0000","last_active":"2024-07-27T15:17:47.766173+0000","last_peered":"2024-07-27T15:17:47.766173+0000","last_clean":"2024-07-27T15:17:47.766173+0000","last_became_active":"2024-07-27T14:42:37.287685+0000","last_became_peered":"2024-07-27T14:42:37.287685+0000","last_unstale":"2024-07-27T15:17:47.766173+0000","last_undegraded":"2024-07-27T15:17:47.766173+0000","last_fullsized":"2024-07-27T15:17:47.766173+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:04:06.389200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":2814,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.360888+0000","last_change":"2024-07-27T14:42:35.360457+0000","last_active":"2024-07-27T15:17:49.360888+0000","last_peered":"2024-07-27T15:17:49.360888+0000","last_clean":"2024-07-27T15:17:49.360888+0000","last_became_active":"2024-07-27T14:42:35.354552+0000","last_became_peered":"2024-07-27T14:42:35.354552+0000","last_unstale":"2024-07-27T15:17:49.360888+0000","last_undegraded":"2024-07-27T15:17:49.360888+0000","last_fullsized":"2024-07-27T15:17:49.360888+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:22:00.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.523171+0000","last_change":"2024-07-27T14:42:34.577783+0000","last_active":"2024-07-27T15:17:48.523171+0000","last_peered":"2024-07-27T15:17:48.523171+0000","last_clean":"2024-07-27T15:17:48.523171+0000","last_became_active":"2024-07-27T14:42:34.568719+0000","last_became_peered":"2024-07-27T14:42:34.568719+0000","last_unstale":"2024-07-27T15:17:48.523171+0000","last_undegraded":"2024-07-27T15:17:48.523171+0000","last_fullsized":"2024-07-27T15:17:48.523171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:14:22.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2929,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.109246+0000","last_change":"2024-07-27T14:41:03.617220+0000","last_active":"2024-07-27T15:17:48.109246+0000","last_peered":"2024-07-27T15:17:48.109246+0000","last_clean":"2024-07-27T15:17:48.109246+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T15:17:48.109246+0000","last_undegraded":"2024-07-27T15:17:48.109246+0000","last_fullsized":"2024-07-27T15:17:48.109246+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:00.649818+0000","last_clean_scrub_stamp":"2024-07-27T14:41:00.649818+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:14:15.632432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":2847,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.738254+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T15:17:47.738254+0000","last_peered":"2024-07-27T15:17:47.738254+0000","last_clean":"2024-07-27T15:17:47.738254+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T15:17:47.738254+0000","last_undegraded":"2024-07-27T15:17:47.738254+0000","last_fullsized":"2024-07-27T15:17:47.738254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:40:29.539136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":2825,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.769897+0000","last_change":"2024-07-27T14:42:37.311409+0000","last_active":"2024-07-27T15:17:47.769897+0000","last_peered":"2024-07-27T15:17:47.769897+0000","last_clean":"2024-07-27T15:17:47.769897+0000","last_became_active":"2024-07-27T14:42:37.189428+0000","last_became_peered":"2024-07-27T14:42:37.189428+0000","last_unstale":"2024-07-27T15:17:47.769897+0000","last_undegraded":"2024-07-27T15:17:47.769897+0000","last_fullsized":"2024-07-27T15:17:47.769897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:15:12.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2846,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.695050+0000","last_change":"2024-07-27T14:41:21.948784+0000","last_active":"2024-07-27T15:17:47.695050+0000","last_peered":"2024-07-27T15:17:47.695050+0000","last_clean":"2024-07-27T15:17:47.695050+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T15:17:47.695050+0000","last_undegraded":"2024-07-27T15:17:47.695050+0000","last_fullsized":"2024-07-27T15:17:47.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:23:48.411987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2828,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.959864+0000","last_change":"2024-07-27T14:42:32.948619+0000","last_active":"2024-07-27T15:17:45.959864+0000","last_peered":"2024-07-27T15:17:45.959864+0000","last_clean":"2024-07-27T15:17:45.959864+0000","last_became_active":"2024-07-27T14:42:32.825779+0000","last_became_peered":"2024-07-27T14:42:32.825779+0000","last_unstale":"2024-07-27T15:17:45.959864+0000","last_undegraded":"2024-07-27T15:17:45.959864+0000","last_fullsized":"2024-07-27T15:17:45.959864+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:44:30.813522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2825,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.506184+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T15:17:48.506184+0000","last_peered":"2024-07-27T15:17:48.506184+0000","last_clean":"2024-07-27T15:17:48.506184+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T15:17:48.506184+0000","last_undegraded":"2024-07-27T15:17:48.506184+0000","last_fullsized":"2024-07-27T15:17:48.506184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2813,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.720308+0000","last_change":"2024-07-27T14:43:03.368313+0000","last_active":"2024-07-27T15:17:47.720308+0000","last_peered":"2024-07-27T15:17:47.720308+0000","last_clean":"2024-07-27T15:17:47.720308+0000","last_became_active":"2024-07-27T14:43:03.360033+0000","last_became_peered":"2024-07-27T14:43:03.360033+0000","last_unstale":"2024-07-27T15:17:47.720308+0000","last_undegraded":"2024-07-27T15:17:47.720308+0000","last_fullsized":"2024-07-27T15:17:47.720308+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:21:29.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":2848,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.678919+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T15:17:47.678919+0000","last_peered":"2024-07-27T15:17:47.678919+0000","last_clean":"2024-07-27T15:17:47.678919+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T15:17:47.678919+0000","last_undegraded":"2024-07-27T15:17:47.678919+0000","last_fullsized":"2024-07-27T15:17:47.678919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:39:58.978046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.704667+0000","last_change":"2024-07-27T14:42:33.420873+0000","last_active":"2024-07-27T15:17:47.704667+0000","last_peered":"2024-07-27T15:17:47.704667+0000","last_clean":"2024-07-27T15:17:47.704667+0000","last_became_active":"2024-07-27T14:42:33.415149+0000","last_became_peered":"2024-07-27T14:42:33.415149+0000","last_unstale":"2024-07-27T15:17:47.704667+0000","last_undegraded":"2024-07-27T15:17:47.704667+0000","last_fullsized":"2024-07-27T15:17:47.704667+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:45:18.005668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2848,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:45.951360+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T15:17:45.951360+0000","last_peered":"2024-07-27T15:17:45.951360+0000","last_clean":"2024-07-27T15:17:45.951360+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T15:17:45.951360+0000","last_undegraded":"2024-07-27T15:17:45.951360+0000","last_fullsized":"2024-07-27T15:17:45.951360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.698776+0000","last_change":"2024-07-27T14:42:57.118705+0000","last_active":"2024-07-27T15:17:47.698776+0000","last_peered":"2024-07-27T15:17:47.698776+0000","last_clean":"2024-07-27T15:17:47.698776+0000","last_became_active":"2024-07-27T14:42:57.112921+0000","last_became_peered":"2024-07-27T14:42:57.112921+0000","last_unstale":"2024-07-27T15:17:47.698776+0000","last_undegraded":"2024-07-27T15:17:47.698776+0000","last_fullsized":"2024-07-27T15:17:47.698776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:02:08.809341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2843,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.771806+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T15:17:47.771806+0000","last_peered":"2024-07-27T15:17:47.771806+0000","last_clean":"2024-07-27T15:17:47.771806+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T15:17:47.771806+0000","last_undegraded":"2024-07-27T15:17:47.771806+0000","last_fullsized":"2024-07-27T15:17:47.771806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:57:17.520382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2815,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.702273+0000","last_change":"2024-07-27T14:42:36.978696+0000","last_active":"2024-07-27T15:17:47.702273+0000","last_peered":"2024-07-27T15:17:47.702273+0000","last_clean":"2024-07-27T15:17:47.702273+0000","last_became_active":"2024-07-27T14:42:36.941428+0000","last_became_peered":"2024-07-27T14:42:36.941428+0000","last_unstale":"2024-07-27T15:17:47.702273+0000","last_undegraded":"2024-07-27T15:17:47.702273+0000","last_fullsized":"2024-07-27T15:17:47.702273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:34:55.684245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2835,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.063984+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T15:17:46.063984+0000","last_peered":"2024-07-27T15:17:46.063984+0000","last_clean":"2024-07-27T15:17:46.063984+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T15:17:46.063984+0000","last_undegraded":"2024-07-27T15:17:46.063984+0000","last_fullsized":"2024-07-27T15:17:46.063984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2819,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.010815+0000","last_change":"2024-07-27T14:42:32.399654+0000","last_active":"2024-07-27T15:17:46.010815+0000","last_peered":"2024-07-27T15:17:46.010815+0000","last_clean":"2024-07-27T15:17:46.010815+0000","last_became_active":"2024-07-27T14:42:32.329977+0000","last_became_peered":"2024-07-27T14:42:32.329977+0000","last_unstale":"2024-07-27T15:17:46.010815+0000","last_undegraded":"2024-07-27T15:17:46.010815+0000","last_fullsized":"2024-07-27T15:17:46.010815+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:17:59.414451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2840,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.777392+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T15:17:47.777392+0000","last_peered":"2024-07-27T15:17:47.777392+0000","last_clean":"2024-07-27T15:17:47.777392+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T15:17:47.777392+0000","last_undegraded":"2024-07-27T15:17:47.777392+0000","last_fullsized":"2024-07-27T15:17:47.777392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:05:24.194010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"44'7","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.354492+0000","last_change":"2024-07-27T14:42:34.811194+0000","last_active":"2024-07-27T15:17:49.354492+0000","last_peered":"2024-07-27T15:17:49.354492+0000","last_clean":"2024-07-27T15:17:49.354492+0000","last_became_active":"2024-07-27T14:42:34.791352+0000","last_became_peered":"2024-07-27T14:42:34.791352+0000","last_unstale":"2024-07-27T15:17:49.354492+0000","last_undegraded":"2024-07-27T15:17:49.354492+0000","last_fullsized":"2024-07-27T15:17:49.354492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:36:47.747804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":2816,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.358672+0000","last_change":"2024-07-27T14:42:35.255013+0000","last_active":"2024-07-27T15:17:49.358672+0000","last_peered":"2024-07-27T15:17:49.358672+0000","last_clean":"2024-07-27T15:17:49.358672+0000","last_became_active":"2024-07-27T14:42:35.248761+0000","last_became_peered":"2024-07-27T14:42:35.248761+0000","last_unstale":"2024-07-27T15:17:49.358672+0000","last_undegraded":"2024-07-27T15:17:49.358672+0000","last_fullsized":"2024-07-27T15:17:49.358672+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:43:59.817053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.059015+0000","last_change":"2024-07-27T14:42:53.137283+0000","last_active":"2024-07-27T15:17:46.059015+0000","last_peered":"2024-07-27T15:17:46.059015+0000","last_clean":"2024-07-27T15:17:46.059015+0000","last_became_active":"2024-07-27T14:42:53.131434+0000","last_became_peered":"2024-07-27T14:42:53.131434+0000","last_unstale":"2024-07-27T15:17:46.059015+0000","last_undegraded":"2024-07-27T15:17:46.059015+0000","last_fullsized":"2024-07-27T15:17:46.059015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:47.643598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2817,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.045645+0000","last_change":"2024-07-27T14:42:32.763809+0000","last_active":"2024-07-27T15:17:46.045645+0000","last_peered":"2024-07-27T15:17:46.045645+0000","last_clean":"2024-07-27T15:17:46.045645+0000","last_became_active":"2024-07-27T14:42:32.755122+0000","last_became_peered":"2024-07-27T14:42:32.755122+0000","last_unstale":"2024-07-27T15:17:46.045645+0000","last_undegraded":"2024-07-27T15:17:46.045645+0000","last_fullsized":"2024-07-27T15:17:46.045645+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:18:44.350815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2805,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.497222+0000","last_change":"2024-07-27T14:42:34.656806+0000","last_active":"2024-07-27T15:17:48.497222+0000","last_peered":"2024-07-27T15:17:48.497222+0000","last_clean":"2024-07-27T15:17:48.497222+0000","last_became_active":"2024-07-27T14:42:34.647534+0000","last_became_peered":"2024-07-27T14:42:34.647534+0000","last_unstale":"2024-07-27T15:17:48.497222+0000","last_undegraded":"2024-07-27T15:17:48.497222+0000","last_fullsized":"2024-07-27T15:17:48.497222+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:43:30.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":2824,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-07-27T15:17:47.774307+0000","last_change":"2024-07-27T14:42:37.982514+0000","last_active":"2024-07-27T15:17:47.774307+0000","last_peered":"2024-07-27T15:17:47.774307+0000","last_clean":"2024-07-27T15:17:47.774307+0000","last_became_active":"2024-07-27T14:42:37.973009+0000","last_became_peered":"2024-07-27T14:42:37.973009+0000","last_unstale":"2024-07-27T15:17:47.774307+0000","last_undegraded":"2024-07-27T15:17:47.774307+0000","last_fullsized":"2024-07-27T15:17:47.774307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:18.889623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2818,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.061508+0000","last_change":"2024-07-27T14:42:32.206159+0000","last_active":"2024-07-27T15:17:46.061508+0000","last_peered":"2024-07-27T15:17:46.061508+0000","last_clean":"2024-07-27T15:17:46.061508+0000","last_became_active":"2024-07-27T14:42:32.194637+0000","last_became_peered":"2024-07-27T14:42:32.194637+0000","last_unstale":"2024-07-27T15:17:46.061508+0000","last_undegraded":"2024-07-27T15:17:46.061508+0000","last_fullsized":"2024-07-27T15:17:46.061508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:04:59.701883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.513378+0000","last_change":"2024-07-27T14:42:30.522614+0000","last_active":"2024-07-27T15:17:48.513378+0000","last_peered":"2024-07-27T15:17:48.513378+0000","last_clean":"2024-07-27T15:17:48.513378+0000","last_became_active":"2024-07-27T14:42:30.517011+0000","last_became_peered":"2024-07-27T14:42:30.517011+0000","last_unstale":"2024-07-27T15:17:48.513378+0000","last_undegraded":"2024-07-27T15:17:48.513378+0000","last_fullsized":"2024-07-27T15:17:48.513378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:36:37.080542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2815,"reported_epoch":1465,"state":"active+clean","last_fresh":"2024-07-27T15:17:46.040712+0000","last_change":"2024-07-27T14:42:44.953045+0000","last_active":"2024-07-27T15:17:46.040712+0000","last_peered":"2024-07-27T15:17:46.040712+0000","last_clean":"2024-07-27T15:17:46.040712+0000","last_became_active":"2024-07-27T14:42:44.426410+0000","last_became_peered":"2024-07-27T14:42:44.426410+0000","last_unstale":"2024-07-27T15:17:46.040712+0000","last_undegraded":"2024-07-27T15:17:46.040712+0000","last_fullsized":"2024-07-27T15:17:46.040712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:38:49.627031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.356862+0000","last_change":"2024-07-27T14:43:09.714646+0000","last_active":"2024-07-27T15:17:49.356862+0000","last_peered":"2024-07-27T15:17:49.356862+0000","last_clean":"2024-07-27T15:17:49.356862+0000","last_became_active":"2024-07-27T14:43:09.707672+0000","last_became_peered":"2024-07-27T14:43:09.707672+0000","last_unstale":"2024-07-27T15:17:49.356862+0000","last_undegraded":"2024-07-27T15:17:49.356862+0000","last_fullsized":"2024-07-27T15:17:49.356862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:09:21.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.367195+0000","last_change":"2024-07-27T14:42:32.528826+0000","last_active":"2024-07-27T15:17:49.367195+0000","last_peered":"2024-07-27T15:17:49.367195+0000","last_clean":"2024-07-27T15:17:49.367195+0000","last_became_active":"2024-07-27T14:42:32.384600+0000","last_became_peered":"2024-07-27T14:42:32.384600+0000","last_unstale":"2024-07-27T15:17:49.367195+0000","last_undegraded":"2024-07-27T15:17:49.367195+0000","last_fullsized":"2024-07-27T15:17:49.367195+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:54:24.285647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2804,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-07-27T15:17:48.502171+0000","last_change":"2024-07-27T14:42:33.230973+0000","last_active":"2024-07-27T15:17:48.502171+0000","last_peered":"2024-07-27T15:17:48.502171+0000","last_clean":"2024-07-27T15:17:48.502171+0000","last_became_active":"2024-07-27T14:42:33.224999+0000","last_became_peered":"2024-07-27T14:42:33.224999+0000","last_unstale":"2024-07-27T15:17:48.502171+0000","last_undegraded":"2024-07-27T15:17:48.502171+0000","last_fullsized":"2024-07-27T15:17:48.502171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:37:06.846510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.349035+0000","last_change":"2024-07-27T14:42:32.891119+0000","last_active":"2024-07-27T15:17:49.349035+0000","last_peered":"2024-07-27T15:17:49.349035+0000","last_clean":"2024-07-27T15:17:49.349035+0000","last_became_active":"2024-07-27T14:42:32.853532+0000","last_became_peered":"2024-07-27T14:42:32.853532+0000","last_unstale":"2024-07-27T15:17:49.349035+0000","last_undegraded":"2024-07-27T15:17:49.349035+0000","last_fullsized":"2024-07-27T15:17:49.349035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:38:00.975708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411770,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76876,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294964,"statfs":{"total":96636764160,"available":96558043136,"internally_reserved":0,"allocated":5570560,"data_stored":2637707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5468,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5599232,"data_stored":2652092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509876,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72720,"kb_used_data":5444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299120,"statfs":{"total":96636764160,"available":96562298880,"internally_reserved":0,"allocated":5574656,"data_stored":2637712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73136,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298704,"statfs":{"total":96636764160,"available":96561872896,"internally_reserved":0,"allocated":5996544,"data_stored":3049927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509882,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72744,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299096,"statfs":{"total":96636764160,"available":96562274304,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76904,"kb_used_data":5464,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294936,"statfs":{"total":96636764160,"available":96558014464,"internally_reserved":0,"allocated":5595136,"data_stored":2652087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575282,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":5832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":5971968,"data_stored":3039624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575285,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72708,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299132,"statfs":{"total":96636764160,"available":96562311168,"internally_reserved":0,"allocated":5554176,"data_stored":2630236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T15:17:52.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-27T15:17:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.464+0000 7f427cfd4640 10 monclient: get_monmap_and_config 2024-07-27T15:17:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.464+0000 7f427cfd4640 10 monclient: build_initial_monmap 2024-07-27T15:17:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.464+0000 7f427cfd4640 10 monclient: monmap: 2024-07-27T15:17:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:52.465438+0000 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:52.465438+0000 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:52.466 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 Processor -- start 2024-07-27T15:17:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 -- start start 2024-07-27T15:17:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient: init 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f427806bbb0 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f4278146d50 0x7f427806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-a con 0x7f4278146d50 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f427813fd90 0x7f4278140240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-b con 0x7f427813fd90 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427806d840 con 0x7f427806bbb0 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f42781461b0 con 0x7f4278146d50 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f427806c7b0 con 0x7f427813fd90 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.465+0000 7f427cfd4640 10 monclient(hunting): authenticate will time out at 3412.491838s 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f427659c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f427806bbb0 0x7f4278146640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:41190/0 (socket says 172.21.15.87:41190) 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f427659c640 1 -- 172.21.15.87:0/4226584255 learned_addr learned my addr 172.21.15.87:0/4226584255 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4275d9b640 1 --2- 172.21.15.87:0/4226584255 >> v2:172.21.15.87:3300/0 conn(0x7f4278146d50 0x7f427806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4276d9d640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f427813fd90 0x7f4278140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4275d9b640 10 monclient(hunting): get_auth_request con 0x7f4278146d50 auth_method 0 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.466+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): get_auth_request con 0x7f427813fd90 auth_method 0 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.467+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 1 -- 172.21.15.87:0/4226584255 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264870122 0 0) 0x7f427806d840 con 0x7f427806bbb0 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 10 monclient(hunting): _negotiate 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 10 monclient(hunting): my global_id is 21886 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.468+0000 7f427559a640 1 -- 172.21.15.87:0/4226584255 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4264003520 con 0x7f427806bbb0 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 10 monclient(hunting): handle_auth_done global_id 14739 payload 274 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 1 -- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f427813fd90 msgr2=0x7f4278140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f427813fd90 0x7f4278140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 1 -- 172.21.15.87:0/4226584255 >> v1:172.21.15.112:6789/0 conn(0x7f427806bbb0 legacy=0x7f4278146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 1 monclient: found mon.noname-a 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.469+0000 7f4275d9b640 1 -- 172.21.15.87:0/4226584255 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4278140780 con 0x7f4278146d50 2024-07-27T15:17:52.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.471+0000 7f4275d9b640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4278146d50 0x7f427806bf90 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f42600097a0 tx=0x7f426002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f363c4cc68b441d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 1 -- 172.21.15.87:0/4226584255 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260035020 con 0x7f4278146d50 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: dump: 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: _finish_auth 0 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: _check_auth_tickets 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 1 -- 172.21.15.87:0/4226584255 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4260031830 con 0x7f4278146d50 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:52.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 1 -- 172.21.15.87:0/4226584255 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260031c00 con 0x7f4278146d50 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427559a640 10 monclient: dump: 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:52.477 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427cfd4640 5 monclient: authenticate success, global_id 14739 2024-07-27T15:17:52.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427cfd4640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:52.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.474+0000 7f427cfd4640 4 set_mon_vals no callback set 2024-07-27T15:17:52.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 10 monclient: shutdown 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 -- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4278146d50 msgr2=0x7f427806bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4278146d50 0x7f427806bf90 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f42600097a0 tx=0x7f426002b8d0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 -- 172.21.15.87:0/4226584255 shutdown_connections 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f427813fd90 0x7f4278140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 --2- 172.21.15.87:0/4226584255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4278146d50 0x7f427806bf90 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 -- 172.21.15.87:0/4226584255 >> 172.21.15.87:0/4226584255 conn(0x7f4278067ee0 msgr2=0x7f42780b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 -- 172.21.15.87:0/4226584255 shutdown_connections 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.475+0000 7f427cfd4640 1 -- 172.21.15.87:0/4226584255 wait complete. 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.476+0000 7f427cfd4640 10 monclient: build_initial_monmap 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.476+0000 7f427cfd4640 10 monclient: monmap: 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.476+0000 7f427cfd4640 1 librados: starting msgr at 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.476+0000 7f427cfd4640 1 librados: starting objecter 2024-07-27T15:17:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.476+0000 7f427cfd4640 1 Processor -- start 2024-07-27T15:17:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 -- start start 2024-07-27T15:17:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 librados: setting wanted keys 2024-07-27T15:17:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 librados: calling monclient init 2024-07-27T15:17:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient: init 2024-07-27T15:17:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 0x7f42781431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-a con 0x7f427806bbb0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 0x7f4278147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-b con 0x7f427813fd90 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4278146d50 0x7f4278147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): picked mon.noname-c con 0x7f4278146d50 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:52.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f42780af4b0 con 0x7f427806bbb0 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42780af330 con 0x7f427813fd90 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f427814d490 con 0x7f4278146d50 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.477+0000 7f427cfd4640 10 monclient(hunting): authenticate will time out at 3412.503594s 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 0x7f42781431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 0x7f4278147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4276d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4278146d50 0x7f4278147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 0x7f42781431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37426/0 (socket says 172.21.15.87:37426) 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 1 -- 172.21.15.87:0/3495716109 learned_addr learned my addr 172.21.15.87:0/3495716109 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): get_auth_request con 0x7f427806bbb0 auth_method 0 2024-07-27T15:17:52.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): get_auth_request con 0x7f427813fd90 auth_method 0 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4276d9d640 10 monclient(hunting): get_auth_request con 0x7f4278146d50 auth_method 0 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4276d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4276d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4276d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.478+0000 7f427659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.479+0000 7f427659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.479+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.479+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.479+0000 7f4276d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 10 monclient(hunting): handle_auth_done global_id 21889 payload 824 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 -- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4278146d50 msgr2=0x7f4278147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4278146d50 0x7f4278147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 -- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 msgr2=0x7f42781431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 0x7f42781431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:52.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:52.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 monclient: found mon.noname-b 2024-07-27T15:17:52.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:52.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:52.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.481+0000 7f4275d9b640 1 -- 172.21.15.87:0/3495716109 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42781db2f0 con 0x7f427813fd90 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.483+0000 7f4275d9b640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 0x7f4278147250 secure :-1 s=READY pgs=5018 cs=0 l=1 rev1=1 crypto rx=0x7f4260033000 tx=0x7f42600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.483+0000 7f426effd640 10 client.?.objecter ms_handle_connect 0x7f427813fd90 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.483+0000 7f426effd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260040070 con 0x7f427813fd90 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: dump: 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.496 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: _finish_auth 0 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426003a930 con 0x7f427813fd90 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260035040 con 0x7f427813fd90 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 5 monclient: authenticate success, global_id 21889 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 10 monclient: _renew_subs 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4278140a20 con 0x7f427813fd90 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f4274d99640 4 set_mon_vals no callback set 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f426effd640 10 monclient: dump: 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 10 client.21889.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 10 monclient: _renew_subs 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42781db9b0 con 0x7f427813fd90 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.487+0000 7f427cfd4640 1 librados: init done 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.488+0000 7f427cfd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.488+0000 7f427cfd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.488+0000 7f427cfd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:52.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.488+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4278148e30 con 0x7f427813fd90 2024-07-27T15:17:52.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.736+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f426004b060 con 0x7f427813fd90 2024-07-27T15:17:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.736+0000 7f426effd640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 0x7f425806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.737+0000 7f427659c640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 0x7f425806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.737+0000 7f427659c640 10 monclient: get_auth_request con 0x7f425806d810 auth_method 0 2024-07-27T15:17:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.737+0000 7f427659c640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 0x7f425806fcd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4270005c00 tx=0x7f4270009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.737+0000 7f426effd640 10 client.21889.objecter ms_handle_connect 0x7f425806d810 2024-07-27T15:17:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.739+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f42600317c0 con 0x7f427813fd90 2024-07-27T15:17:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.739+0000 7f426effd640 10 client.21889.objecter ms_dispatch 0x7f4278000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.739+0000 7f426effd640 3 client.21889.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.739+0000 7f426effd640 3 client.21889.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.739+0000 7f426effd640 20 client.21889.objecter dump_active .. 0 homeless 2024-07-27T15:17:52.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.750+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f4278148e30 con 0x7f427813fd90 2024-07-27T15:17:52.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.750+0000 7f426effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:52.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.750+0000 7f426effd640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.938+0000 7f427cfd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-27T15:17:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.938+0000 7f427cfd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T15:17:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.938+0000 7f427cfd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.938+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4278140240 con 0x7f427813fd90 2024-07-27T15:17:52.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.994+0000 7f426effd640 1 -- 172.21.15.87:0/3495716109 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1468) ==== 74+0+17789 (secure 0 0 0) 0x7f426007abb0 con 0x7f427813fd90 2024-07-27T15:17:52.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.994+0000 7f426effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-27T15:17:52.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.994+0000 7f426effd640 10 monclient: _finish_command 2 = system:0 2024-07-27T15:17:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:17:52.997 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":1468,"fsid":"a5c605d8-a191-47d5-9a8c-d25753c56cf8","created":"2024-07-27T14:40:14.023779+0000","modified":"2024-07-27T15:17:49.234588+0000","last_up_change":"2024-07-27T14:41:10.123887+0000","last_in_change":"2024-07-27T14:40:19.591859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":118,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T14:41:00.102594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-27T14:41:16.750108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-07-27T14:42:26.915638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f32f839-eba6-44b8-aee5-21988b13d1f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6809","nonce":3385608780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6811","nonce":3385608780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6815","nonce":3385608780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3385608780},{"type":"v1","addr":"172.21.15.87:6813","nonce":3385608780}]},"public_addr":"172.21.15.87:6809/3385608780","cluster_addr":"172.21.15.87:6811/3385608780","heartbeat_back_addr":"172.21.15.87:6815/3385608780","heartbeat_front_addr":"172.21.15.87:6813/3385608780","state":["exists","up"]},{"osd":1,"uuid":"ddadaf3f-715f-4ea5-80de-b3266c50cbf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1450,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6801","nonce":1509783986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6803","nonce":1509783986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6807","nonce":1509783986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1509783986},{"type":"v1","addr":"172.21.15.87:6805","nonce":1509783986}]},"public_addr":"172.21.15.87:6801/1509783986","cluster_addr":"172.21.15.87:6803/1509783986","heartbeat_back_addr":"172.21.15.87:6807/1509783986","heartbeat_front_addr":"172.21.15.87:6805/1509783986","state":["exists","up"]},{"osd":2,"uuid":"5276af43-fb52-45c0-b5f8-35e0ed1499e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1457,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6822","nonce":357294290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6825","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6827","nonce":357294290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6831","nonce":357294290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":357294290},{"type":"v1","addr":"172.21.15.87:6829","nonce":357294290}]},"public_addr":"172.21.15.87:6822/357294290","cluster_addr":"172.21.15.87:6827/357294290","heartbeat_back_addr":"172.21.15.87:6831/357294290","heartbeat_front_addr":"172.21.15.87:6829/357294290","state":["exists","up"]},{"osd":3,"uuid":"a14626c8-e29d-4fb7-aec6-179ac404e813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6817","nonce":1100053525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6819","nonce":1100053525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6826","nonce":1100053525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":1100053525},{"type":"v1","addr":"172.21.15.87:6823","nonce":1100053525}]},"public_addr":"172.21.15.87:6817/1100053525","cluster_addr":"172.21.15.87:6819/1100053525","heartbeat_back_addr":"172.21.15.87:6826/1100053525","heartbeat_front_addr":"172.21.15.87:6823/1100053525","state":["exists","up"]},{"osd":4,"uuid":"a73fab50-80f0-4936-a1f4-91b6fd224114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6827","nonce":1578415927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6829","nonce":1578415927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6833","nonce":1578415927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":1578415927},{"type":"v1","addr":"172.21.15.112:6831","nonce":1578415927}]},"public_addr":"172.21.15.112:6827/1578415927","cluster_addr":"172.21.15.112:6829/1578415927","heartbeat_back_addr":"172.21.15.112:6833/1578415927","heartbeat_front_addr":"172.21.15.112:6831/1578415927","state":["exists","up"]},{"osd":5,"uuid":"ae27ce3a-6a6a-4ee1-a32a-ccc4f6994560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6811","nonce":3751677016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6813","nonce":3751677016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6817","nonce":3751677016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3751677016},{"type":"v1","addr":"172.21.15.112:6815","nonce":3751677016}]},"public_addr":"172.21.15.112:6811/3751677016","cluster_addr":"172.21.15.112:6813/3751677016","heartbeat_back_addr":"172.21.15.112:6817/3751677016","heartbeat_front_addr":"172.21.15.112:6815/3751677016","state":["exists","up"]},{"osd":6,"uuid":"4b2984bd-8611-4a93-863a-852d8366fc3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6803","nonce":591715754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6805","nonce":591715754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6809","nonce":591715754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":591715754},{"type":"v1","addr":"172.21.15.112:6807","nonce":591715754}]},"public_addr":"172.21.15.112:6803/591715754","cluster_addr":"172.21.15.112:6805/591715754","heartbeat_back_addr":"172.21.15.112:6809/591715754","heartbeat_front_addr":"172.21.15.112:6807/591715754","state":["exists","up"]},{"osd":7,"uuid":"c6c9e332-d670-435d-a37b-d6232ca7009c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6819","nonce":2667521237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6821","nonce":2667521237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6825","nonce":2667521237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2667521237},{"type":"v1","addr":"172.21.15.112:6823","nonce":2667521237}]},"public_addr":"172.21.15.112:6819/2667521237","cluster_addr":"172.21.15.112:6821/2667521237","heartbeat_back_addr":"172.21.15.112:6825/2667521237","heartbeat_front_addr":"172.21.15.112:6823/2667521237","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.359413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.864798+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.468303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.219576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.792415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.790276+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:58.126847+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T14:40:57.527378+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":3}]},{"pool":19,"snaps":[{"begin":5,"length":2}]},{"pool":23,"snaps":[{"begin":1,"length":1}]},{"pool":78,"snaps":[{"begin":1,"length":1}]},{"pool":87,"snaps":[{"begin":7,"length":2}]},{"pool":92,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":1,"length":2}]},{"pool":106,"snaps":[{"begin":14,"length":1}]},{"pool":107,"snaps":[{"begin":3,"length":1}]},{"pool":109,"snaps":[{"begin":3,"length":2}]},{"pool":110,"snaps":[{"begin":2,"length":2}]},{"pool":120,"snaps":[{"begin":2,"length":3}]},{"pool":123,"snaps":[{"begin":4,"length":2}]},{"pool":134,"snaps":[{"begin":2,"length":2}]},{"pool":144,"snaps":[{"begin":2,"length":1}]},{"pool":147,"snaps":[{"begin":3,"length":2}]},{"pool":156,"snaps":[{"begin":4,"length":4}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":194,"snaps":[{"begin":7,"length":1}]},{"pool":201,"snaps":[{"begin":8,"length":1}]},{"pool":203,"snaps":[{"begin":10,"length":1}]},{"pool":204,"snaps":[{"begin":11,"length":2}]},{"pool":207,"snaps":[{"begin":14,"length":1}]},{"pool":250,"snaps":[{"begin":2,"length":1}]},{"pool":251,"snaps":[{"begin":3,"length":1}]},{"pool":254,"snaps":[{"begin":4,"length":1}]},{"pool":258,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 10 librados: watch_flush enter 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 10 librados: watch_flush exit 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 20 client.21889.objecter shutdown clearing up homeless session... 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 10 client.21889.objecter successfully canceled tick 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 msgr2=0x7f425806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 0x7f425806fcd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4270005c00 tx=0x7f4270009040 comp rx=0 tx=0).stop 2024-07-27T15:17:52.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 10 monclient: shutdown 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 msgr2=0x7f4278147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 0x7f4278147250 secure :-1 s=READY pgs=5018 cs=0 l=1 rev1=1 crypto rx=0x7f4260033000 tx=0x7f42600046c0 comp rx=0 tx=0).stop 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 shutdown_connections 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f425806d810 0x7f425806fcd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4278146d50 0x7f4278147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f427813fd90 0x7f4278147250 unknown :-1 s=CLOSED pgs=5018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 --2- 172.21.15.87:0/3495716109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f427806bbb0 0x7f42781431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 >> 172.21.15.87:0/3495716109 conn(0x7f4278067ee0 msgr2=0x7f42780b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 shutdown_connections 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 -- 172.21.15.87:0/3495716109 wait complete. 2024-07-27T15:17:53.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:52.997+0000 7f427cfd4640 1 librados: shutdown 2024-07-27T15:17:54.011 INFO:tasks.ceph:Scrubbing osd.0 2024-07-27T15:17:54.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:54.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 10 monclient: get_monmap_and_config 2024-07-27T15:17:54.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 10 monclient: monmap: 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:54.114253+0000 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:54.114253+0000 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:54.115 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 1 Processor -- start 2024-07-27T15:17:54.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 1 -- start start 2024-07-27T15:17:54.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.113+0000 7f42d0a92640 10 monclient: init 2024-07-27T15:17:54.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-07-27T15:17:54.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 0x7f42cc0aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc0b1b80 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 0x7f42cc13c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-b con 0x7f42cc0ad340 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42cc13ce70 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42cc0ad080 con 0x7f42cc13ce70 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f42cc0b1f60 con 0x7f42cc0b1b80 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc0ae430 con 0x7f42cc0ad340 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.114+0000 7f42d0a92640 10 monclient(hunting): authenticate will time out at 3414.140634s 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42c9d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 0x7f42cc13c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42ca59c640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 0x7f42cc0aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42cad9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f42cc13ce70 0x7f42cc142c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52898/0 (socket says 172.21.15.87:52898) 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42cad9d640 1 -- 172.21.15.87:0/3124444067 learned_addr learned my addr 172.21.15.87:0/3124444067 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42c9d9b640 10 monclient(hunting): get_auth_request con 0x7f42cc0ad340 auth_method 0 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42c9d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42ca59c640 10 monclient(hunting): get_auth_request con 0x7f42cc0b1b80 auth_method 0 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42ca59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.115+0000 7f42ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.116+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 1 -- 172.21.15.87:0/3124444067 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131375667 0 0) 0x7f42cc0ad080 con 0x7f42cc13ce70 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 10 monclient(hunting): _negotiate 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 10 monclient(hunting): my global_id is 21892 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c959a640 1 -- 172.21.15.87:0/3124444067 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b8005730 con 0x7f42cc13ce70 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_done global_id 15503 payload 274 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 1 -- 172.21.15.87:0/3124444067 >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 msgr2=0x7f42cc0aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 1 --2- 172.21.15.87:0/3124444067 >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 0x7f42cc0aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 1 -- 172.21.15.87:0/3124444067 >> v1:172.21.15.112:6789/0 conn(0x7f42cc13ce70 legacy=0x7f42cc142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 1 monclient: found mon.noname-b 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42c9d9b640 1 -- 172.21.15.87:0/3124444067 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc0ac420 con 0x7f42cc0ad340 2024-07-27T15:17:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.117+0000 7f42ca59c640 1 --2- 172.21.15.87:0/3124444067 >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 0x7f42cc0aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.119+0000 7f42c9d9b640 1 --2- 172.21.15.87:0/3124444067 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 0x7f42cc13c930 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0097a0 tx=0x7f42bc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19c73914e5cc9b2a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 1 -- 172.21.15.87:0/3124444067 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0315a0 con 0x7f42cc0ad340 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: dump: 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 1 -- 172.21.15.87:0/3124444067 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc037070 con 0x7f42cc0ad340 2024-07-27T15:17:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 5 monclient: authenticate success, global_id 15503 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 1 -- 172.21.15.87:0/3124444067 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc031960 con 0x7f42cc0ad340 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 4 set_mon_vals no callback set 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42c959a640 10 monclient: dump: 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 10 monclient: shutdown 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 1 -- 172.21.15.87:0/3124444067 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 msgr2=0x7f42cc13c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.123+0000 7f42d0a92640 1 --2- 172.21.15.87:0/3124444067 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 0x7f42cc13c930 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0097a0 tx=0x7f42bc02b840 comp rx=0 tx=0).stop 2024-07-27T15:17:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 -- 172.21.15.87:0/3124444067 shutdown_connections 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 --2- 172.21.15.87:0/3124444067 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc0ad340 0x7f42cc13c930 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 --2- 172.21.15.87:0/3124444067 >> v2:172.21.15.87:3300/0 conn(0x7f42cc0b1b80 0x7f42cc0aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 -- 172.21.15.87:0/3124444067 >> 172.21.15.87:0/3124444067 conn(0x7f42cc0b4d30 msgr2=0x7f42cc0b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 -- 172.21.15.87:0/3124444067 shutdown_connections 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 1 -- 172.21.15.87:0/3124444067 wait complete. 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.124+0000 7f42d0a92640 10 monclient: monmap: 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 librados: starting msgr at 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 librados: starting objecter 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 Processor -- start 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 -- start start 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 librados: setting wanted keys 2024-07-27T15:17:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 librados: calling monclient init 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient: init 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 0x7f42cc140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-b con 0x7f42cc0b1b80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc13ce70 0x7f42cc0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-c con 0x7f42cc13ce70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42cc072df0 0x7f42cc1d3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc072df0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc143590 con 0x7f42cc072df0 2024-07-27T15:17:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc143410 con 0x7f42cc0b1b80 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc143710 con 0x7f42cc13ce70 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.125+0000 7f42d0a92640 10 monclient(hunting): authenticate will time out at 3414.151944s 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 0x7f42cc140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42c9d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc13ce70 0x7f42cc0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42cad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42cc072df0 0x7f42cc1d3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 0x7f42cc140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45326/0 (socket says 172.21.15.87:45326) 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 1 -- 172.21.15.87:0/2218187396 learned_addr learned my addr 172.21.15.87:0/2218187396 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 10 monclient(hunting): get_auth_request con 0x7f42cc0b1b80 auth_method 0 2024-07-27T15:17:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42c9d9b640 10 monclient(hunting): get_auth_request con 0x7f42cc13ce70 auth_method 0 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42c9d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42cad9d640 10 monclient(hunting): get_auth_request con 0x7f42cc072df0 auth_method 0 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42cad9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.126+0000 7f42cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.127+0000 7f42cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 10 monclient(hunting): handle_auth_done global_id 21895 payload 824 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 -- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc13ce70 msgr2=0x7f42cc0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42cc13ce70 0x7f42cc0728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 -- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42cc072df0 msgr2=0x7f42cc1d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42cc072df0 0x7f42cc1d3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 monclient: found mon.noname-b 2024-07-27T15:17:54.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:54.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.130+0000 7f42ca59c640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc06fe60 con 0x7f42cc0b1b80 2024-07-27T15:17:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.144+0000 7f42ca59c640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 0x7f42cc140030 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ebe0 tx=0x7f42b400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.144+0000 7f42c2ffd640 10 client.?.objecter ms_handle_connect 0x7f42cc0b1b80 2024-07-27T15:17:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.144+0000 7f42c2ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:54.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4017070 con 0x7f42cc0b1b80 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: dump: 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b400cc30 con 0x7f42cc0b1b80 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 5 monclient: authenticate success, global_id 21895 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 10 monclient: _renew_subs 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42cc070140 con 0x7f42cc0b1b80 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4013470 con 0x7f42cc0b1b80 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c2ffd640 10 monclient: dump: 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.157 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42c8d99640 4 set_mon_vals no callback set 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 10 client.21895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 10 monclient: _renew_subs 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42cc073280 con 0x7f42cc0b1b80 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.153+0000 7f42d0a92640 1 librados: init done 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 10 client.21895.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 15 client.21895.objecter _session_command_op_assign -1 1 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 20 client.21895.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:54.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 10 client.21895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:54.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:54.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.154+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f42cc06efb0 con 0x7f42cc0b1b80 2024-07-27T15:17:54.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.172+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f42b401a070 con 0x7f42cc0b1b80 2024-07-27T15:17:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.173+0000 7f42c2ffd640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f42ac06d810 0x7f42ac06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.173+0000 7f42c9d9b640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f42ac06d810 0x7f42ac06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.173+0000 7f42c9d9b640 10 monclient: get_auth_request con 0x7f42ac06d810 auth_method 0 2024-07-27T15:17:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.173+0000 7f42c2ffd640 10 client.21895.objecter ms_handle_connect 0x7f42ac06d810 2024-07-27T15:17:54.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.173+0000 7f42c9d9b640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f42ac06d810 0x7f42ac06fcd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f42bc036000 tx=0x7f42bc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f42b4057890 con 0x7f42cc0b1b80 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 10 client.21895.objecter ms_dispatch 0x7f42cc000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 3 client.21895.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 3 client.21895.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 10 client.21895.objecter checking command 1 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f42ac075580 0x7f42ac077a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter _get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 15 client.21895.objecter _session_command_op_remove -1 1 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter _get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 15 client.21895.objecter _session_command_op_assign 0 1 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 4 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 10 client.21895.objecter _send_command 1 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42b40093e0 con 0x7f42ac075580 2024-07-27T15:17:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42c2ffd640 20 client.21895.objecter dump_active .. 0 homeless 2024-07-27T15:17:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.175+0000 7f42cad9d640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f42ac075580 0x7f42ac077a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.176+0000 7f42cad9d640 10 monclient: get_auth_request con 0x7f42ac075580 auth_method 0 2024-07-27T15:17:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.176+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7f42b4055a90 con 0x7f42cc0b1b80 2024-07-27T15:17:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.176+0000 7f42c2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.179+0000 7f42cad9d640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f42ac075580 0x7f42ac077a20 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.179+0000 7f42c2ffd640 10 client.21895.objecter ms_handle_connect 0x7f42ac075580 2024-07-27T15:17:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.202+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== osd.0 v2:172.21.15.87:6808/3385608780 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42b40093e0 con 0x7f42ac075580 2024-07-27T15:17:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.202+0000 7f42c2ffd640 10 client.21895.objecter ms_dispatch 0x7f42cc000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.202+0000 7f42c2ffd640 10 client.21895.objecter _finish_command 1 = system:0 2024-07-27T15:17:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.202+0000 7f42c2ffd640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 4 2024-07-27T15:17:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.202+0000 7f42c2ffd640 15 client.21895.objecter _session_command_op_remove 0 1 2024-07-27T15:17:54.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 10 client.21895.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 15 client.21895.objecter _session_command_op_assign -1 2 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 20 client.21895.objecter _get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 20 client.21895.objecter _get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 15 client.21895.objecter _session_command_op_remove -1 2 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 20 client.21895.objecter get_session s=0x7f42ac074eb0 osd=0 3 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 15 client.21895.objecter _session_command_op_assign 0 2 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 4 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 10 client.21895.objecter _send_command 2 2024-07-27T15:17:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.215+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 --> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f42cc13c930 con 0x7f42ac075580 2024-07-27T15:17:54.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.228+0000 7f42c2ffd640 1 -- 172.21.15.87:0/2218187396 <== osd.0 v2:172.21.15.87:6808/3385608780 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f42cc13c930 con 0x7f42ac075580 2024-07-27T15:17:54.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.228+0000 7f42c2ffd640 10 client.21895.objecter ms_dispatch 0x7f42cc000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:54.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.228+0000 7f42c2ffd640 10 client.21895.objecter _finish_command 2 = system:0 2024-07-27T15:17:54.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.228+0000 7f42c2ffd640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 4 2024-07-27T15:17:54.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.228+0000 7f42c2ffd640 15 client.21895.objecter _session_command_op_remove 0 2 2024-07-27T15:17:54.230 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:54.230 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:54.230 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:54.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 10 librados: watch_flush enter 2024-07-27T15:17:54.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 10 librados: watch_flush exit 2024-07-27T15:17:54.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 10 client.21895.objecter close_session for osd.0 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f42ac075580 msgr2=0x7f42ac077a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.87:6808/3385608780,v1:172.21.15.87:6809/3385608780] conn(0x7f42ac075580 0x7f42ac077a20 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 20 client.21895.objecter put_session s=0x7f42ac074eb0 osd=0 1 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 20 client.21895.objecter shutdown clearing up homeless session... 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 10 client.21895.objecter successfully canceled tick 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f42ac06d810 msgr2=0x7f42ac06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f42ac06d810 0x7f42ac06fcd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f42bc036000 tx=0x7f42bc03d040 comp rx=0 tx=0).stop 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 10 monclient: shutdown 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:54.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 msgr2=0x7f42cc140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 --2- 172.21.15.87:0/2218187396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42cc0b1b80 0x7f42cc140030 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ebe0 tx=0x7f42b400c520 comp rx=0 tx=0).stop 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42cad9d640 1 -- 172.21.15.87:0/2218187396 reap_dead start 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 shutdown_connections 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 >> 172.21.15.87:0/2218187396 conn(0x7f42cc0b4d30 msgr2=0x7f42cc0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 shutdown_connections 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 -- 172.21.15.87:0/2218187396 wait complete. 2024-07-27T15:17:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.229+0000 7f42d0a92640 1 librados: shutdown 2024-07-27T15:17:54.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-07-27T15:17:54.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.338+0000 7f197f5c4640 10 monclient: get_monmap_and_config 2024-07-27T15:17:54.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.338+0000 7f197f5c4640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.338+0000 7f197f5c4640 10 monclient: monmap: 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:54.339603+0000 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:54.339603+0000 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:54.340 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 1 Processor -- start 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 1 -- start start 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient: init 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1978075090 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1978075470 0x7f19780b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-c con 0x7f1978075470 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 0x7f197814d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-b con 0x7f19781460d0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.339+0000 7f197f5c4640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1978070450 con 0x7f1978075090 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f1978070200 con 0x7f1978075470 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f19780b1630 con 0x7f19781460d0 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197f5c4640 10 monclient(hunting): authenticate will time out at 3414.366026s 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197db61640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 0x7f197814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197cb5f640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1978075470 0x7f19780b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197d360640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1978075090 0x7f1978073050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52900/0 (socket says 172.21.15.87:52900) 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197d360640 1 -- 172.21.15.87:0/3908179353 learned_addr learned my addr 172.21.15.87:0/3908179353 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197db61640 10 monclient(hunting): get_auth_request con 0x7f19781460d0 auth_method 0 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197db61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197db61640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.340+0000 7f197db61640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): get_auth_request con 0x7f1978075470 auth_method 0 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.341+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 10 monclient(hunting): handle_auth_done global_id 15509 payload 274 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 1 -- 172.21.15.87:0/3908179353 >> v2:172.21.15.112:3300/0 conn(0x7f1978075470 msgr2=0x7f19780b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 1 --2- 172.21.15.87:0/3908179353 >> v2:172.21.15.112:3300/0 conn(0x7f1978075470 0x7f19780b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 1 -- 172.21.15.87:0/3908179353 >> v1:172.21.15.112:6789/0 conn(0x7f1978075090 legacy=0x7f1978073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 1 monclient: found mon.noname-b 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.343+0000 7f197db61640 1 -- 172.21.15.87:0/3908179353 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19780b0860 con 0x7f19781460d0 2024-07-27T15:17:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.344+0000 7f197db61640 1 --2- 172.21.15.87:0/3908179353 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 0x7f197814d460 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f1978070380 tx=0x7f197002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42daa641367436 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 1 -- 172.21.15.87:0/3908179353 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970039070 con 0x7f19781460d0 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 10 monclient: dump: 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.346+0000 7f196ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.347+0000 7f197f5c4640 5 monclient: authenticate success, global_id 15509 2024-07-27T15:17:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.347+0000 7f197f5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:54.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 1 -- 172.21.15.87:0/3908179353 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197002cbb0 con 0x7f19781460d0 2024-07-27T15:17:54.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 1 -- 172.21.15.87:0/3908179353 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970034440 con 0x7f19781460d0 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f196ffff640 10 monclient: dump: 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.353 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 4 set_mon_vals no callback set 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 10 monclient: shutdown 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 -- 172.21.15.87:0/3908179353 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 msgr2=0x7f197814d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 --2- 172.21.15.87:0/3908179353 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 0x7f197814d460 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f1978070380 tx=0x7f197002be00 comp rx=0 tx=0).stop 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 -- 172.21.15.87:0/3908179353 shutdown_connections 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 --2- 172.21.15.87:0/3908179353 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f19781460d0 0x7f197814d460 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 --2- 172.21.15.87:0/3908179353 >> v2:172.21.15.112:3300/0 conn(0x7f1978075470 0x7f19780b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 -- 172.21.15.87:0/3908179353 >> 172.21.15.87:0/3908179353 conn(0x7f19780b1870 msgr2=0x7f19780b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 -- 172.21.15.87:0/3908179353 shutdown_connections 2024-07-27T15:17:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.348+0000 7f197f5c4640 1 -- 172.21.15.87:0/3908179353 wait complete. 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 10 monclient: monmap: 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 librados: starting msgr at 2024-07-27T15:17:54.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 librados: starting objecter 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 Processor -- start 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 -- start start 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 librados: setting wanted keys 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 1 librados: calling monclient init 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.349+0000 7f197f5c4640 10 monclient: init 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1978075090 0x7f19781414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-b con 0x7f1978075090 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19781460d0 0x7f1978142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-a con 0x7f19781460d0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): picked mon.noname-c con 0x7f1978140950 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f197806f330 con 0x7f19781460d0 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f197806f1b0 con 0x7f1978075090 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f1978146520 con 0x7f1978140950 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197f5c4640 10 monclient(hunting): authenticate will time out at 3414.376486s 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.350+0000 7f197db61640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197cb5f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19781460d0 0x7f1978142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1978075090 0x7f19781414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37088/0 (socket says 172.21.15.87:37088) 2024-07-27T15:17:54.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 1 -- 172.21.15.87:0/1376680885 learned_addr learned my addr 172.21.15.87:0/1376680885 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): get_auth_request con 0x7f1978140950 auth_method 0 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): get_auth_request con 0x7f1978075090 auth_method 0 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197cb5f640 10 monclient(hunting): get_auth_request con 0x7f19781460d0 auth_method 0 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197cb5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197cb5f640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197cb5f640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197db61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.351+0000 7f197d360640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.352+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.352+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.352+0000 7f197cb5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 10 monclient(hunting): handle_auth_done global_id 15512 payload 824 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 -- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1978075090 msgr2=0x7f19781414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1978075090 0x7f19781414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 -- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19781460d0 msgr2=0x7f1978142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19781460d0 0x7f1978142010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 monclient: found mon.noname-c 2024-07-27T15:17:54.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.354+0000 7f197db61640 1 -- 172.21.15.87:0/1376680885 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1978142550 con 0x7f1978140950 2024-07-27T15:17:54.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.356+0000 7f197db61640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f197002c410 tx=0x7f1970004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.356+0000 7f196dffb640 10 client.?.objecter ms_handle_connect 0x7f1978140950 2024-07-27T15:17:54.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.356+0000 7f196dffb640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970039050 con 0x7f1978140950 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: dump: 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.362 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1970045070 con 0x7f1978140950 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197003e830 con 0x7f1978140950 2024-07-27T15:17:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196dffb640 10 monclient: dump: 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 5 monclient: authenticate success, global_id 15512 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 10 monclient: _renew_subs 2024-07-27T15:17:54.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19781d8620 con 0x7f1978140950 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f196f7fe640 4 set_mon_vals no callback set 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 10 client.15512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 10 monclient: _renew_subs 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19781d8b20 con 0x7f1978140950 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.360+0000 7f197f5c4640 1 librados: init done 2024-07-27T15:17:54.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.361+0000 7f197f5c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:54.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.361+0000 7f197f5c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:54.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.361+0000 7f197f5c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:54.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.361+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19781439c0 con 0x7f1978140950 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.379+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f197003e9d0 con 0x7f1978140950 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.379+0000 7f196dffb640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 0x7f195c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.380+0000 7f197d360640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 0x7f195c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.380+0000 7f197d360640 10 monclient: get_auth_request con 0x7f195c06d7c0 auth_method 0 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.380+0000 7f197d360640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 0x7f195c06fc80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1974007fd0 tx=0x7f1974007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.380+0000 7f196dffb640 10 client.15512.objecter ms_handle_connect 0x7f195c06d7c0 2024-07-27T15:17:54.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.383+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f197007a9e0 con 0x7f1978140950 2024-07-27T15:17:54.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.383+0000 7f196dffb640 10 client.15512.objecter ms_dispatch 0x7f1978000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:54.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.383+0000 7f196dffb640 3 client.15512.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:54.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.383+0000 7f196dffb640 3 client.15512.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:54.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.383+0000 7f196dffb640 20 client.15512.objecter dump_active .. 0 homeless 2024-07-27T15:17:54.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.623+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f19781439c0 con 0x7f1978140950 2024-07-27T15:17:54.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.623+0000 7f196dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:54.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.623+0000 7f196dffb640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:54.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.804+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f19780b0d80 con 0x7f195c06d7c0 2024-07-27T15:17:54.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.805+0000 7f196dffb640 1 -- 172.21.15.87:0/1376680885 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f19780b0d80 con 0x7f195c06d7c0 2024-07-27T15:17:54.806 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 0 to deep-scrub 2024-07-27T15:17:54.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.806+0000 7f197f5c4640 10 librados: watch_flush enter 2024-07-27T15:17:54.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 10 librados: watch_flush exit 2024-07-27T15:17:54.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 20 client.15512.objecter shutdown clearing up homeless session... 2024-07-27T15:17:54.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 10 client.15512.objecter successfully canceled tick 2024-07-27T15:17:54.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 msgr2=0x7f195c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 0x7f195c06fc80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1974007fd0 tx=0x7f1974007e80 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 10 monclient: shutdown 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 msgr2=0x7f19781d80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f197002c410 tx=0x7f1970004a40 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 shutdown_connections 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f195c06d7c0 0x7f195c06fc80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1978140950 0x7f19781d80e0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19781460d0 0x7f1978142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 --2- 172.21.15.87:0/1376680885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1978075090 0x7f19781414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 >> 172.21.15.87:0/1376680885 conn(0x7f19780b1870 msgr2=0x7f197806c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:54.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 shutdown_connections 2024-07-27T15:17:54.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 -- 172.21.15.87:0/1376680885 wait complete. 2024-07-27T15:17:54.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.807+0000 7f197f5c4640 1 librados: shutdown 2024-07-27T15:17:54.819 INFO:tasks.ceph:Scrubbing osd.1 2024-07-27T15:17:54.820 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.930+0000 7fab10d85640 10 monclient: get_monmap_and_config 2024-07-27T15:17:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.930+0000 7fab10d85640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.930+0000 7fab10d85640 10 monclient: monmap: 2024-07-27T15:17:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:54.931481+0000 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:54.931481+0000 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:54.932 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 Processor -- start 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 -- start start 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient: init 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 0x7fab0c0b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c0b5d60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab0c13ff60 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:54.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fab0c1466d0 0x7fab0c146b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-c con 0x7fab0c1466d0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab0c142cc0 con 0x7fab0c13ff60 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fab0c0b6140 con 0x7fab0c1466d0 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fab0c146240 con 0x7fab0c0b5d60 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.931+0000 7fab10d85640 10 monclient(hunting): authenticate will time out at 3414.957844s 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.932+0000 7fab01d9b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fab0c13ff60 0x7fab0c140360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:58590/0 (socket says 172.21.15.87:58590) 2024-07-27T15:17:54.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.932+0000 7fab01d9b640 1 -- 172.21.15.87:0/1735583421 learned_addr learned my addr 172.21.15.87:0/1735583421 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 921586041 0 0) 0x7fab0c142cc0 con 0x7fab0c13ff60 2024-07-27T15:17:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 10 monclient(hunting): _negotiate 2024-07-27T15:17:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 10 monclient(hunting): my global_id is 14754 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.934+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaf8003520 con 0x7fab0c13ff60 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.937+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 703103057 0 0) 0x7faaf8003520 con 0x7fab0c13ff60 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.937+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 msgr2=0x7fab0c0b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.937+0000 7fab09d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 0x7fab0c0b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.937+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 >> v2:172.21.15.112:3300/0 conn(0x7fab0c1466d0 msgr2=0x7fab0c146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.937+0000 7fab09d9b640 1 --2- 172.21.15.87:0/1735583421 >> v2:172.21.15.112:3300/0 conn(0x7fab0c1466d0 0x7fab0c146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 monclient: found mon.noname-a-legacy 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.87:6789/0 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c0aa9c0 con 0x7fab0c13ff60 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1560156229 0 0) 0x7faaf4001670 con 0x7fab0c13ff60 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: dump: 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.945 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] but i'm connected to v1:172.21.15.87:6789/0, reconnecting 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 >> v1:172.21.15.87:6789/0 conn(0x7fab0c13ff60 legacy=0x7fab0c140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf80039b0 0x7faaf8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): picked mon.b con 0x7faaf80039b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faaf8004800 0x7faaf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): picked mon.a con 0x7faaf8004800 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 0x7faaf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): picked mon.c con 0x7fab0c0b5d60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c0b6140 con 0x7faaf8004800 2024-07-27T15:17:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c146240 con 0x7faaf80039b0 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7faaf8005680 con 0x7fab0c0b5d60 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.938+0000 7fab09d9b640 10 monclient(hunting): _finish_auth 0 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.940+0000 7fab01d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faaf8004800 0x7faaf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): get_auth_request con 0x7faaf8004800 auth_method 0 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.941+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T15:17:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 10 monclient(hunting): handle_auth_done global_id 14754 payload 293 2024-07-27T15:17:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 -- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 msgr2=0x7faaf800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab0c0b5d60 0x7faaf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 -- 172.21.15.87:0/1735583421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf80039b0 msgr2=0x7faaf8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf80039b0 0x7faaf8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 monclient: found mon.a 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 1 -- 172.21.15.87:0/1735583421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf8003520 con 0x7faaf8004800 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab10d85640 5 monclient: authenticate success, global_id 14754 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.943+0000 7fab10d85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.945+0000 7fab01d9b640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faaf8004800 0x7faaf8004c80 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7faaf400c190 tx=0x7faaf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405800b421cd98a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4007c80 con 0x7faaf8004800 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: dump: 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab10d85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf4030070 con 0x7faaf8004800 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab10d85640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 1 -- 172.21.15.87:0/1735583421 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4005db0 con 0x7faaf8004800 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab10d85640 4 set_mon_vals no callback set 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.948+0000 7fab09d9b640 10 monclient: dump: 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 10 monclient: shutdown 2024-07-27T15:17:54.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 -- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faaf8004800 msgr2=0x7faaf8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 --2- 172.21.15.87:0/1735583421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faaf8004800 0x7faaf8004c80 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7faaf400c190 tx=0x7faaf8005830 comp rx=0 tx=0).stop 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab0ad9d640 1 -- 172.21.15.87:0/1735583421 reap_dead start 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 -- 172.21.15.87:0/1735583421 shutdown_connections 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 -- 172.21.15.87:0/1735583421 >> 172.21.15.87:0/1735583421 conn(0x7fab0c067a50 msgr2=0x7fab0c067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 -- 172.21.15.87:0/1735583421 shutdown_connections 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.949+0000 7fab10d85640 1 -- 172.21.15.87:0/1735583421 wait complete. 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 10 monclient: build_initial_monmap 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 10 monclient: monmap: 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:54.954 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 librados: starting msgr at 2024-07-27T15:17:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 librados: starting objecter 2024-07-27T15:17:54.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 Processor -- start 2024-07-27T15:17:54.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 -- start start 2024-07-27T15:17:54.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 librados: setting wanted keys 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 1 librados: calling monclient init 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.950+0000 7fab10d85640 10 monclient: init 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faaf8004800 0x7fab0c0acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-c con 0x7faaf8004800 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 0x7fab0c1dbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c0ad990 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab0c1dc180 0x7fab0c0b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c1dc180 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c0aae00 con 0x7fab0c0ad990 2024-07-27T15:17:54.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c0aac80 con 0x7fab0c1dc180 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fab0c0b4db0 con 0x7faaf8004800 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab10d85640 10 monclient(hunting): authenticate will time out at 3414.977521s 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.951+0000 7fab01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 0x7fab0c1dbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 0x7fab0c1dbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33490/0 (socket says 172.21.15.87:33490) 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 1 -- 172.21.15.87:0/2621250965 learned_addr learned my addr 172.21.15.87:0/2621250965 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): get_auth_request con 0x7fab0c0ad990 auth_method 0 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.952+0000 7fab01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 10 monclient(hunting): handle_auth_done global_id 14757 payload 824 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 -- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faaf8004800 msgr2=0x7fab0c0acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faaf8004800 0x7fab0c0acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 -- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab0c1dc180 msgr2=0x7fab0c0b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab0c1dc180 0x7fab0c0b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 monclient: found mon.noname-a 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.955+0000 7fab01d9b640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c0aea20 con 0x7fab0c0ad990 2024-07-27T15:17:54.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.958+0000 7fab01d9b640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 0x7fab0c1dbc40 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7faaf400c280 tx=0x7faaf4001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.958+0000 7fab037fe640 10 client.?.objecter ms_handle_connect 0x7fab0c0ad990 2024-07-27T15:17:54.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.958+0000 7fab037fe640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:54.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf40049f0 con 0x7fab0c0ad990 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: dump: 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: _finish_auth 0 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: _check_auth_tickets 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf4045070 con 0x7fab0c0ad990 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004dd0 con 0x7fab0c0ad990 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab037fe640 10 monclient: dump: 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab10d85640 5 monclient: authenticate success, global_id 14757 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab10d85640 10 monclient: _renew_subs 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab10d85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c0aed00 con 0x7fab0c0ad990 2024-07-27T15:17:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.961+0000 7fab0959a640 4 set_mon_vals no callback set 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.962+0000 7fab10d85640 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.962+0000 7fab10d85640 10 monclient: _renew_subs 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.962+0000 7fab10d85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.962+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c0addb0 con 0x7fab0c0ad990 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.962+0000 7fab10d85640 1 librados: init done 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 10 client.14757.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 15 client.14757.objecter _session_command_op_assign -1 1 2024-07-27T15:17:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 20 client.14757.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.963+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab0c146b40 con 0x7fab0c0ad990 2024-07-27T15:17:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.983+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faaf4007070 con 0x7fab0c0ad990 2024-07-27T15:17:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.984+0000 7fab037fe640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faaec06d810 0x7faaec06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.984+0000 7fab0a59c640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faaec06d810 0x7faaec06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.984+0000 7fab0a59c640 10 monclient: get_auth_request con 0x7faaec06d810 auth_method 0 2024-07-27T15:17:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.985+0000 7fab037fe640 10 client.14757.objecter ms_handle_connect 0x7faaec06d810 2024-07-27T15:17:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.985+0000 7fab0a59c640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faaec06d810 0x7faaec06fcd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faafc009710 tx=0x7faafc006d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7faaf4079670 con 0x7fab0c0ad990 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 10 client.14757.objecter ms_dispatch 0x7fab0c000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 3 client.14757.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 3 client.14757.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 10 client.14757.objecter checking command 1 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7faaec075580 0x7faaec077a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter _get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:54.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 15 client.14757.objecter _session_command_op_remove -1 1 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter _get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 15 client.14757.objecter _session_command_op_assign 1 1 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 4 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 10 client.14757.objecter _send_command 1 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaf403ce50 con 0x7faaec075580 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 20 client.14757.objecter dump_active .. 0 homeless 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7faaf4077870 con 0x7fab0c0ad990 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.988+0000 7fab037fe640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:54.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.989+0000 7fab0ad9d640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7faaec075580 0x7faaec077a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:54.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.989+0000 7fab0ad9d640 10 monclient: get_auth_request con 0x7faaec075580 auth_method 0 2024-07-27T15:17:54.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.992+0000 7fab037fe640 10 client.14757.objecter ms_handle_connect 0x7faaec075580 2024-07-27T15:17:54.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:54.992+0000 7fab0ad9d640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7faaec075580 0x7faaec077a20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.015+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== osd.1 v2:172.21.15.87:6800/1509783986 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faaf403ce50 con 0x7faaec075580 2024-07-27T15:17:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.015+0000 7fab037fe640 10 client.14757.objecter ms_dispatch 0x7fab0c000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.015+0000 7fab037fe640 10 client.14757.objecter _finish_command 1 = system:0 2024-07-27T15:17:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.015+0000 7fab037fe640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 4 2024-07-27T15:17:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.015+0000 7fab037fe640 15 client.14757.objecter _session_command_op_remove 1 1 2024-07-27T15:17:55.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 10 client.14757.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:55.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 15 client.14757.objecter _session_command_op_assign -1 2 2024-07-27T15:17:55.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 20 client.14757.objecter _get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:55.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 20 client.14757.objecter _get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 15 client.14757.objecter _session_command_op_remove -1 2 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 20 client.14757.objecter get_session s=0x7faaec074eb0 osd=1 3 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 15 client.14757.objecter _session_command_op_assign 1 2 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 4 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 10 client.14757.objecter _send_command 2 2024-07-27T15:17:55.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.028+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 --> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fab0c144a30 con 0x7faaec075580 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab037fe640 1 -- 172.21.15.87:0/2621250965 <== osd.1 v2:172.21.15.87:6800/1509783986 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fab0c144a30 con 0x7faaec075580 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab037fe640 10 client.14757.objecter ms_dispatch 0x7fab0c000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab037fe640 10 client.14757.objecter _finish_command 2 = system:0 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab037fe640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 4 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab037fe640 15 client.14757.objecter _session_command_op_remove 1 2 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:55.042 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:55.043 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 10 librados: watch_flush enter 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 10 librados: watch_flush exit 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 10 client.14757.objecter close_session for osd.1 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7faaec075580 msgr2=0x7faaec077a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:6800/1509783986,v1:172.21.15.87:6801/1509783986] conn(0x7faaec075580 0x7faaec077a20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 20 client.14757.objecter put_session s=0x7faaec074eb0 osd=1 1 2024-07-27T15:17:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 20 client.14757.objecter shutdown clearing up homeless session... 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 10 client.14757.objecter successfully canceled tick 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faaec06d810 msgr2=0x7faaec06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7faaec06d810 0x7faaec06fcd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faafc009710 tx=0x7faafc006d10 comp rx=0 tx=0).stop 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 10 monclient: shutdown 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 msgr2=0x7fab0c1dbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 --2- 172.21.15.87:0/2621250965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab0c0ad990 0x7fab0c1dbc40 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7faaf400c280 tx=0x7faaf4001970 comp rx=0 tx=0).stop 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab0ad9d640 1 -- 172.21.15.87:0/2621250965 reap_dead start 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 shutdown_connections 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 >> 172.21.15.87:0/2621250965 conn(0x7fab0c067a50 msgr2=0x7fab0c145510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 shutdown_connections 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 -- 172.21.15.87:0/2621250965 wait complete. 2024-07-27T15:17:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.041+0000 7fab10d85640 1 librados: shutdown 2024-07-27T15:17:55.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.153+0000 7f5906405640 10 monclient: get_monmap_and_config 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.153+0000 7f5906405640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.153+0000 7f5906405640 10 monclient: monmap: 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.155 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:55.154922+0000 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:55.154922+0000 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:55.156 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:55.157 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:55.157 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.154+0000 7f5906405640 1 Processor -- start 2024-07-27T15:17:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.154+0000 7f5906405640 1 -- start start 2024-07-27T15:17:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.154+0000 7f5906405640 10 monclient: init 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f59000b1be0 0x7f5900144340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): picked mon.noname-a con 0x7f59000b1be0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5900073560 0x7f59000739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): picked mon.noname-b con 0x7f5900073560 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f59001408b0 0x7f5900140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): picked mon.noname-c con 0x7f59001408b0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f590006e780 con 0x7f59000b1be0 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f590006e9f0 con 0x7f59001408b0 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f590006ecd0 con 0x7f5900073560 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.155+0000 7f5906405640 10 monclient(hunting): authenticate will time out at 3415.181457s 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5900073560 0x7f59000739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58fffff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f59000b1be0 0x7f5900144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f59001408b0 0x7f5900140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f59001408b0 0x7f5900140ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45362/0 (socket says 172.21.15.87:45362) 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 1 -- 172.21.15.87:0/2490059566 learned_addr learned my addr 172.21.15.87:0/2490059566 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): get_auth_request con 0x7f59001408b0 auth_method 0 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58fffff640 10 monclient(hunting): get_auth_request con 0x7f59000b1be0 auth_method 0 2024-07-27T15:17:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58fffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58fffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): get_auth_request con 0x7f5900073560 auth_method 0 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.156+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.157+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.157+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.157+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.157+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 10 monclient(hunting): handle_auth_done global_id 21904 payload 274 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 -- 172.21.15.87:0/2490059566 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5900073560 msgr2=0x7f59000739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 --2- 172.21.15.87:0/2490059566 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5900073560 0x7f59000739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 -- 172.21.15.87:0/2490059566 >> v2:172.21.15.87:3300/0 conn(0x7f59000b1be0 msgr2=0x7f5900144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 --2- 172.21.15.87:0/2490059566 >> v2:172.21.15.87:3300/0 conn(0x7f59000b1be0 0x7f5900144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 monclient: found mon.noname-c 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.158+0000 7f59049a2640 1 -- 172.21.15.87:0/2490059566 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59001449a0 con 0x7f59001408b0 2024-07-27T15:17:55.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.160+0000 7f59049a2640 1 --2- 172.21.15.87:0/2490059566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f5900140ce0 secure :-1 s=READY pgs=5025 cs=0 l=1 rev1=1 crypto rx=0x7f59000aaaf0 tx=0x7f58f802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abdc2a64d758adac server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 1 -- 172.21.15.87:0/2490059566 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f802c9e0 con 0x7f59001408b0 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: dump: 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f5906405640 5 monclient: authenticate success, global_id 21904 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 1 -- 172.21.15.87:0/2490059566 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f802cb80 con 0x7f59001408b0 2024-07-27T15:17:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f5906405640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f5906405640 4 set_mon_vals no callback set 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 1 -- 172.21.15.87:0/2490059566 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f80343f0 con 0x7f59001408b0 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.164+0000 7f58feffd640 10 monclient: dump: 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.168 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 10 monclient: shutdown 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 -- 172.21.15.87:0/2490059566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 msgr2=0x7f5900140ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 --2- 172.21.15.87:0/2490059566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f5900140ce0 secure :-1 s=READY pgs=5025 cs=0 l=1 rev1=1 crypto rx=0x7f59000aaaf0 tx=0x7f58f802be40 comp rx=0 tx=0).stop 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 -- 172.21.15.87:0/2490059566 shutdown_connections 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 --2- 172.21.15.87:0/2490059566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f5900140ce0 unknown :-1 s=CLOSED pgs=5025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 --2- 172.21.15.87:0/2490059566 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5900073560 0x7f59000739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 --2- 172.21.15.87:0/2490059566 >> v2:172.21.15.87:3300/0 conn(0x7f59000b1be0 0x7f5900144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 -- 172.21.15.87:0/2490059566 >> 172.21.15.87:0/2490059566 conn(0x7f59000b4d30 msgr2=0x7f59000b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 -- 172.21.15.87:0/2490059566 shutdown_connections 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 1 -- 172.21.15.87:0/2490059566 wait complete. 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.165+0000 7f5906405640 10 monclient: monmap: 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 librados: starting msgr at 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 librados: starting objecter 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 Processor -- start 2024-07-27T15:17:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 -- start start 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 librados: setting wanted keys 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 1 librados: calling monclient init 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.166+0000 7f5906405640 10 monclient: init 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5900073560 0x7f590006f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): picked mon.noname-a con 0x7f5900073560 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000b1be0 0x7f590013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): picked mon.noname-c con 0x7f59000b1be0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): picked mon.noname-b con 0x7f59001408b0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f59001413a0 con 0x7f5900073560 2024-07-27T15:17:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5900141220 con 0x7f59001408b0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f5900141520 con 0x7f59000b1be0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f5906405640 10 monclient(hunting): authenticate will time out at 3415.193206s 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f58fffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5900073560 0x7f590006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f58ff7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000b1be0 0x7f590013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.167+0000 7f59049a2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45364/0 (socket says 172.21.15.87:45364) 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 1 -- 172.21.15.87:0/779573832 learned_addr learned my addr 172.21.15.87:0/779573832 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): get_auth_request con 0x7f59001408b0 auth_method 0 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): get_auth_request con 0x7f59000b1be0 auth_method 0 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): get_auth_request con 0x7f5900073560 auth_method 0 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f59049a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.168+0000 7f58fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 10 monclient(hunting): handle_auth_done global_id 21907 payload 824 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 -- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000b1be0 msgr2=0x7f590013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000b1be0 0x7f590013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 -- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5900073560 msgr2=0x7f590006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5900073560 0x7f590006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 monclient: found mon.noname-b 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.171+0000 7f59049a2640 1 -- 172.21.15.87:0/779573832 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590006fa40 con 0x7f59001408b0 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.173+0000 7f59049a2640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 secure :-1 s=READY pgs=5026 cs=0 l=1 rev1=1 crypto rx=0x7f58f802c450 tx=0x7f58f8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.173+0000 7f58fcff9640 10 client.?.objecter ms_handle_connect 0x7f59001408b0 2024-07-27T15:17:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.173+0000 7f58fcff9640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:55.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8042020 con 0x7f59001408b0 2024-07-27T15:17:55.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:55.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: dump: 2024-07-27T15:17:55.179 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: _finish_auth 0 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: _check_auth_tickets 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:55.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f803e450 con 0x7f59001408b0 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 5 monclient: authenticate success, global_id 21907 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 10 monclient: _renew_subs 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8009510 con 0x7f59001408b0 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590006ff40 con 0x7f59001408b0 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fcff9640 10 monclient: dump: 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.181 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f58fe7fc640 4 set_mon_vals no callback set 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 10 client.21907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 10 monclient: _renew_subs 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59001d9be0 con 0x7f59001408b0 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.176+0000 7f5906405640 1 librados: init done 2024-07-27T15:17:55.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.178+0000 7f5906405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:55.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.178+0000 7f5906405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:55.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.178+0000 7f5906405640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.178+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f590004fc60 con 0x7f59001408b0 2024-07-27T15:17:55.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.199+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f58f8045020 con 0x7f59001408b0 2024-07-27T15:17:55.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.200+0000 7f58fcff9640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 0x7f58e006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.200+0000 7f58fffff640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 0x7f58e006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.200+0000 7f58fffff640 10 monclient: get_auth_request con 0x7f58e006d810 auth_method 0 2024-07-27T15:17:55.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.200+0000 7f58fcff9640 10 client.21907.objecter ms_handle_connect 0x7f58e006d810 2024-07-27T15:17:55.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.200+0000 7f58fffff640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 0x7f58e006fcd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009ac0 tx=0x7f58f0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.202+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f58f8079e30 con 0x7f59001408b0 2024-07-27T15:17:55.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.202+0000 7f58fcff9640 10 client.21907.objecter ms_dispatch 0x7f5900000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:55.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.202+0000 7f58fcff9640 3 client.21907.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:55.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.202+0000 7f58fcff9640 3 client.21907.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:55.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.202+0000 7f58fcff9640 20 client.21907.objecter dump_active .. 0 homeless 2024-07-27T15:17:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.445+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f590004fc60 con 0x7f59001408b0 2024-07-27T15:17:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.445+0000 7f58fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.445+0000 7f58fcff9640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:55.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.634+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5900144340 con 0x7f58e006d810 2024-07-27T15:17:55.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.635+0000 7f58fcff9640 1 -- 172.21.15.87:0/779573832 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5900144340 con 0x7f58e006d810 2024-07-27T15:17:55.636 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 1 to deep-scrub 2024-07-27T15:17:55.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 10 librados: watch_flush enter 2024-07-27T15:17:55.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 10 librados: watch_flush exit 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 20 client.21907.objecter shutdown clearing up homeless session... 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 10 client.21907.objecter successfully canceled tick 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 msgr2=0x7f58e006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 0x7f58e006fcd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009ac0 tx=0x7f58f0009190 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 10 monclient: shutdown 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 msgr2=0x7f59001d96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 secure :-1 s=READY pgs=5026 cs=0 l=1 rev1=1 crypto rx=0x7f58f802c450 tx=0x7f58f8004a40 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 shutdown_connections 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f58e006d810 0x7f58e006fcd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59001408b0 0x7f59001d96a0 unknown :-1 s=CLOSED pgs=5026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000b1be0 0x7f590013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 --2- 172.21.15.87:0/779573832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5900073560 0x7f590006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 >> 172.21.15.87:0/779573832 conn(0x7f59000b4d30 msgr2=0x7f59000af000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 shutdown_connections 2024-07-27T15:17:55.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 -- 172.21.15.87:0/779573832 wait complete. 2024-07-27T15:17:55.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.637+0000 7f5906405640 1 librados: shutdown 2024-07-27T15:17:55.648 INFO:tasks.ceph:Scrubbing osd.2 2024-07-27T15:17:55.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:55.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.749+0000 7f3546910640 10 monclient: get_monmap_and_config 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.749+0000 7f3546910640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.749+0000 7f3546910640 10 monclient: monmap: 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:55.750741+0000 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:55.750741+0000 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:55.752 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.750+0000 7f3546910640 1 Processor -- start 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.750+0000 7f3546910640 1 -- start start 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.750+0000 7f3546910640 10 monclient: init 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-07-27T15:17:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f35400b1b80 0x7f35400aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): picked mon.noname-c con 0x7f35400b1b80 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:55.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35400ad3f0 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:55.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35400ac600 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:55.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35400ae630 con 0x7f35400ac600 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35400ac0e0 con 0x7f35400ad3f0 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f3540139bc0 con 0x7f35400b1b80 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3546910640 10 monclient(hunting): authenticate will time out at 3415.777266s 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3544ead640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f35400ac600 0x7f3540142c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:58604/0 (socket says 172.21.15.87:58604) 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.751+0000 7f3544ead640 1 -- 172.21.15.87:0/3490499553 learned_addr learned my addr 172.21.15.87:0/3490499553 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.752+0000 7f353ffff640 1 --2- 172.21.15.87:0/3490499553 >> v2:172.21.15.112:3300/0 conn(0x7f35400b1b80 0x7f35400aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.752+0000 7f353ffff640 10 monclient(hunting): get_auth_request con 0x7f35400b1b80 auth_method 0 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.752+0000 7f353ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.752+0000 7f353ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.752+0000 7f353ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.753+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.753+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.753+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484024996 0 0) 0x7f35400ae630 con 0x7f35400ac600 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 10 monclient(hunting): _negotiate 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 10 monclient(hunting): my global_id is 14766 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.754+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3528005730 con 0x7f35400ac600 2024-07-27T15:17:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352097345 0 0) 0x7f35400ac0e0 con 0x7f35400ad3f0 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 10 monclient(hunting): _negotiate 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 10 monclient(hunting): my global_id is 21913 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.755+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35400ae630 con 0x7f35400ad3f0 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 10 monclient(hunting): handle_auth_done global_id 21910 payload 274 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 1 -- 172.21.15.87:0/3490499553 >> v1:172.21.15.112:6789/0 conn(0x7f35400ad3f0 legacy=0x7f35401394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 1 -- 172.21.15.87:0/3490499553 >> v1:172.21.15.87:6789/0 conn(0x7f35400ac600 legacy=0x7f3540142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 1 monclient: found mon.noname-c 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.756+0000 7f353ffff640 1 -- 172.21.15.87:0/3490499553 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3540143380 con 0x7f35400b1b80 2024-07-27T15:17:55.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.759+0000 7f353ffff640 1 --2- 172.21.15.87:0/3490499553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35400aea50 secure :-1 s=READY pgs=5028 cs=0 l=1 rev1=1 crypto rx=0x7f35300097a0 tx=0x7f353002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acca92499b6ca42d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530034070 con 0x7f35400b1b80 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: dump: 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: _finish_auth 0 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530030610 con 0x7f35400b1b80 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 1 -- 172.21.15.87:0/3490499553 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35300309e0 con 0x7f35400b1b80 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 5 monclient: authenticate success, global_id 21910 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 4 set_mon_vals no callback set 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f353effd640 10 monclient: dump: 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 10 monclient: shutdown 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 -- 172.21.15.87:0/3490499553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 msgr2=0x7f35400aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 --2- 172.21.15.87:0/3490499553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35400aea50 secure :-1 s=READY pgs=5028 cs=0 l=1 rev1=1 crypto rx=0x7f35300097a0 tx=0x7f353002b960 comp rx=0 tx=0).stop 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 -- 172.21.15.87:0/3490499553 shutdown_connections 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 --2- 172.21.15.87:0/3490499553 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35400aea50 unknown :-1 s=CLOSED pgs=5028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 -- 172.21.15.87:0/3490499553 >> 172.21.15.87:0/3490499553 conn(0x7f35400b4d30 msgr2=0x7f35400b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 -- 172.21.15.87:0/3490499553 shutdown_connections 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 -- 172.21.15.87:0/3490499553 wait complete. 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 10 monclient: monmap: 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.771 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 librados: starting msgr at 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.765+0000 7f3546910640 1 librados: starting objecter 2024-07-27T15:17:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 Processor -- start 2024-07-27T15:17:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 -- start start 2024-07-27T15:17:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 librados: setting wanted keys 2024-07-27T15:17:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 librados: calling monclient init 2024-07-27T15:17:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient: init 2024-07-27T15:17:55.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T15:17:55.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35400ac600 0x7f354006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): picked mon.noname-a con 0x7f35400ac600 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35400ad3f0 0x7f35401d4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): picked mon.noname-c con 0x7f35400ad3f0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35401d4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): picked mon.noname-b con 0x7f35400b1b80 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f35400aa870 con 0x7f35400ac600 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35400aa6f0 con 0x7f35400b1b80 2024-07-27T15:17:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f35400aa9f0 con 0x7f35400ad3f0 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.766+0000 7f3546910640 10 monclient(hunting): authenticate will time out at 3415.792832s 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35400ac600 0x7f354006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353f7fe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35400ad3f0 0x7f35401d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35401d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35401d4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45388/0 (socket says 172.21.15.87:45388) 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 1 -- 172.21.15.87:0/738225795 learned_addr learned my addr 172.21.15.87:0/738225795 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 10 monclient(hunting): get_auth_request con 0x7f35400b1b80 auth_method 0 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f3544ead640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353ffff640 10 monclient(hunting): get_auth_request con 0x7f35400ac600 auth_method 0 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353f7fe640 10 monclient(hunting): get_auth_request con 0x7f35400ad3f0 auth_method 0 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.767+0000 7f353f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f3544ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f3544ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f3544ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.768+0000 7f353f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 10 monclient(hunting): handle_auth_done global_id 21916 payload 824 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 -- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35400ad3f0 msgr2=0x7f35401d4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35400ad3f0 0x7f35401d4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 -- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35400ac600 msgr2=0x7f354006f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35400ac600 0x7f354006f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 monclient: found mon.noname-b 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.770+0000 7f3544ead640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35400712c0 con 0x7f35400b1b80 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.773+0000 7f3544ead640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35401d4550 secure :-1 s=READY pgs=5029 cs=0 l=1 rev1=1 crypto rx=0x7f353800cb70 tx=0x7f353800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.773+0000 7f353cff9640 10 client.?.objecter ms_handle_connect 0x7f35400b1b80 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.773+0000 7f353cff9640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538017070 con 0x7f35400b1b80 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 10 monclient: dump: 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 10 monclient: _finish_auth 0 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 10 monclient: _check_auth_tickets 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353800ad60 con 0x7f35400b1b80 2024-07-27T15:17:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f3546910640 5 monclient: authenticate success, global_id 21916 2024-07-27T15:17:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f3546910640 10 monclient: _renew_subs 2024-07-27T15:17:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.776+0000 7f3546910640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35401d4a90 con 0x7f35400b1b80 2024-07-27T15:17:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353cff9640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538012640 con 0x7f35400b1b80 2024-07-27T15:17:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353cff9640 10 monclient: dump: 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f353e7fc640 4 set_mon_vals no callback set 2024-07-27T15:17:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 client.21916.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 monclient: _renew_subs 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354006f680 con 0x7f35400b1b80 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 1 librados: init done 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 client.21916.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 15 client.21916.objecter _session_command_op_assign -1 1 2024-07-27T15:17:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 20 client.21916.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 client.21916.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.777+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3540070120 con 0x7f35400b1b80 2024-07-27T15:17:55.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.797+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f353801a070 con 0x7f35400b1b80 2024-07-27T15:17:55.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.798+0000 7f353cff9640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f352c06d810 0x7f352c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.798+0000 7f353ffff640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f352c06d810 0x7f352c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.798+0000 7f353ffff640 10 monclient: get_auth_request con 0x7f352c06d810 auth_method 0 2024-07-27T15:17:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.798+0000 7f353cff9640 10 client.21916.objecter ms_handle_connect 0x7f352c06d810 2024-07-27T15:17:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.798+0000 7f353ffff640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f352c06d810 0x7f352c06fcd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3530036000 tx=0x7f353003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.801+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f3538057720 con 0x7f35400b1b80 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.801+0000 7f353cff9640 10 client.21916.objecter ms_dispatch 0x7f3540000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.801+0000 7f353cff9640 3 client.21916.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.801+0000 7f353cff9640 3 client.21916.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 10 client.21916.objecter checking command 1 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f352c075580 0x7f352c077a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter _get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 15 client.21916.objecter _session_command_op_remove -1 1 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter _get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 15 client.21916.objecter _session_command_op_assign 2 1 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 4 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 10 client.21916.objecter _send_command 1 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3538016ec0 con 0x7f352c075580 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 20 client.21916.objecter dump_active .. 0 homeless 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7f3538055920 con 0x7f35400b1b80 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353cff9640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.802+0000 7f353f7fe640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f352c075580 0x7f352c077a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.803+0000 7f353f7fe640 10 monclient: get_auth_request con 0x7f352c075580 auth_method 0 2024-07-27T15:17:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.805+0000 7f353f7fe640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f352c075580 0x7f352c077a20 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.805+0000 7f353cff9640 10 client.21916.objecter ms_handle_connect 0x7f352c075580 2024-07-27T15:17:55.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.829+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== osd.2 v2:172.21.15.87:6820/357294290 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3538016ec0 con 0x7f352c075580 2024-07-27T15:17:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.829+0000 7f353cff9640 10 client.21916.objecter ms_dispatch 0x7f3540000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.829+0000 7f353cff9640 10 client.21916.objecter _finish_command 1 = system:0 2024-07-27T15:17:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.829+0000 7f353cff9640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 4 2024-07-27T15:17:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.829+0000 7f353cff9640 15 client.21916.objecter _session_command_op_remove 2 1 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 10 client.21916.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 15 client.21916.objecter _session_command_op_assign -1 2 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 20 client.21916.objecter _get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 20 client.21916.objecter _get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 15 client.21916.objecter _session_command_op_remove -1 2 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 20 client.21916.objecter get_session s=0x7f352c074eb0 osd=2 3 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 15 client.21916.objecter _session_command_op_assign 2 2 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 4 2024-07-27T15:17:55.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 10 client.21916.objecter _send_command 2 2024-07-27T15:17:55.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.843+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 --> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f35400aea50 con 0x7f352c075580 2024-07-27T15:17:55.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f353cff9640 1 -- 172.21.15.87:0/738225795 <== osd.2 v2:172.21.15.87:6820/357294290 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f35400aea50 con 0x7f352c075580 2024-07-27T15:17:55.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f353cff9640 10 client.21916.objecter ms_dispatch 0x7f3540000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:55.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f353cff9640 10 client.21916.objecter _finish_command 2 = system:0 2024-07-27T15:17:55.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f353cff9640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 4 2024-07-27T15:17:55.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f353cff9640 15 client.21916.objecter _session_command_op_remove 2 2 2024-07-27T15:17:55.861 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:55.861 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:55.861 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:55.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 10 librados: watch_flush enter 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 10 librados: watch_flush exit 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 10 client.21916.objecter close_session for osd.2 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f352c075580 msgr2=0x7f352c077a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.87:6820/357294290,v1:172.21.15.87:6822/357294290] conn(0x7f352c075580 0x7f352c077a20 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 20 client.21916.objecter put_session s=0x7f352c074eb0 osd=2 1 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 20 client.21916.objecter shutdown clearing up homeless session... 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.859+0000 7f3546910640 10 client.21916.objecter successfully canceled tick 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f352c06d810 msgr2=0x7f352c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f352c06d810 0x7f352c06fcd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3530036000 tx=0x7f353003b040 comp rx=0 tx=0).stop 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 10 monclient: shutdown 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 msgr2=0x7f35401d4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 --2- 172.21.15.87:0/738225795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35400b1b80 0x7f35401d4550 secure :-1 s=READY pgs=5029 cs=0 l=1 rev1=1 crypto rx=0x7f353800cb70 tx=0x7f353800a430 comp rx=0 tx=0).stop 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3544ead640 1 -- 172.21.15.87:0/738225795 reap_dead start 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 shutdown_connections 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 >> 172.21.15.87:0/738225795 conn(0x7f35400b4d30 msgr2=0x7f35400b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 shutdown_connections 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 -- 172.21.15.87:0/738225795 wait complete. 2024-07-27T15:17:55.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.860+0000 7f3546910640 1 librados: shutdown 2024-07-27T15:17:55.871 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-07-27T15:17:55.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.971+0000 7f4776f53640 10 monclient: get_monmap_and_config 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.971+0000 7f4776f53640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.971+0000 7f4776f53640 10 monclient: monmap: 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:55.972901+0000 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:55.972901+0000 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.973 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:55.974 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:55.974 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:55.974 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:55.974 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:55.974 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.972+0000 7f4776f53640 1 Processor -- start 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.972+0000 7f4776f53640 1 -- start start 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.972+0000 7f4776f53640 10 monclient: init 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f47700678a0 0x7f4770071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-a con 0x7f47700678a0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4770140980 0x7f47700721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-c con 0x7f4770140980 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f47700afb00 0x7f477013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-b con 0x7f47700afb00 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f4770067c80 con 0x7f47700678a0 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f4770073870 con 0x7f4770140980 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f47700b1f70 con 0x7f47700afb00 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4776f53640 10 monclient(hunting): authenticate will time out at 3415.999346s 2024-07-27T15:17:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f47754f0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f47700afb00 0x7f477013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f4774cef640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f47700678a0 0x7f4770071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.973+0000 7f476ffff640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4770140980 0x7f47700721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4770140980 0x7f47700721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45390/0 (socket says 172.21.15.87:45390) 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 1 -- 172.21.15.87:0/504014918 learned_addr learned my addr 172.21.15.87:0/504014918 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): get_auth_request con 0x7f4770140980 auth_method 0 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): get_auth_request con 0x7f47700678a0 auth_method 0 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): get_auth_request con 0x7f47700afb00 auth_method 0 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.974+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 10 monclient(hunting): handle_auth_done global_id 21919 payload 274 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 -- 172.21.15.87:0/504014918 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f47700afb00 msgr2=0x7f477013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 --2- 172.21.15.87:0/504014918 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f47700afb00 0x7f477013dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 -- 172.21.15.87:0/504014918 >> v2:172.21.15.87:3300/0 conn(0x7f47700678a0 msgr2=0x7f4770071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 --2- 172.21.15.87:0/504014918 >> v2:172.21.15.87:3300/0 conn(0x7f47700678a0 0x7f4770071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 monclient: found mon.noname-c 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.976+0000 7f476ffff640 1 -- 172.21.15.87:0/504014918 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4770073c90 con 0x7f4770140980 2024-07-27T15:17:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.978+0000 7f476ffff640 1 --2- 172.21.15.87:0/504014918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4770140980 0x7f47700721e0 secure :-1 s=READY pgs=5030 cs=0 l=1 rev1=1 crypto rx=0x7f4770068710 tx=0x7f475c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6560e1d5e475c4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 1 -- 172.21.15.87:0/504014918 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c035020 con 0x7f4770140980 2024-07-27T15:17:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: dump: 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.983 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: _finish_auth 0 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: _check_auth_tickets 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 1 -- 172.21.15.87:0/504014918 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c031870 con 0x7f4770140980 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 1 -- 172.21.15.87:0/504014918 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c031c40 con 0x7f4770140980 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f4776f53640 5 monclient: authenticate success, global_id 21919 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f4776f53640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f4776f53640 4 set_mon_vals no callback set 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.981+0000 7f476f7fe640 10 monclient: dump: 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:55.985 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 10 monclient: shutdown 2024-07-27T15:17:55.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:55.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 -- 172.21.15.87:0/504014918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4770140980 msgr2=0x7f47700721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 --2- 172.21.15.87:0/504014918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4770140980 0x7f47700721e0 secure :-1 s=READY pgs=5030 cs=0 l=1 rev1=1 crypto rx=0x7f4770068710 tx=0x7f475c02b910 comp rx=0 tx=0).stop 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 -- 172.21.15.87:0/504014918 shutdown_connections 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 --2- 172.21.15.87:0/504014918 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f47700afb00 0x7f477013dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 --2- 172.21.15.87:0/504014918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4770140980 0x7f47700721e0 unknown :-1 s=CLOSED pgs=5030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 --2- 172.21.15.87:0/504014918 >> v2:172.21.15.87:3300/0 conn(0x7f47700678a0 0x7f4770071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 -- 172.21.15.87:0/504014918 >> 172.21.15.87:0/504014918 conn(0x7f47700b5920 msgr2=0x7f47700b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 -- 172.21.15.87:0/504014918 shutdown_connections 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.982+0000 7f4776f53640 1 -- 172.21.15.87:0/504014918 wait complete. 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.983+0000 7f4776f53640 10 monclient: build_initial_monmap 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.983+0000 7f4776f53640 10 monclient: monmap: 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:55.990 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.983+0000 7f4776f53640 1 librados: starting msgr at 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.983+0000 7f4776f53640 1 librados: starting objecter 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.983+0000 7f4776f53640 1 Processor -- start 2024-07-27T15:17:55.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 -- start start 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 librados: setting wanted keys 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 librados: calling monclient init 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient: init 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-b con 0x7f47700678a0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47700afb00 0x7f4770136840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-a con 0x7f47700afb00 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4770140980 0x7f4770136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): picked mon.noname-c con 0x7f4770140980 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f477013e580 con 0x7f47700afb00 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f477013e400 con 0x7f47700678a0 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f47700b0350 con 0x7f4770140980 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.984+0000 7f4776f53640 10 monclient(hunting): authenticate will time out at 3416.010755s 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f47754f0640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4770140980 0x7f4770136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f476ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47700afb00 0x7f4770136840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45398/0 (socket says 172.21.15.87:45398) 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 1 -- 172.21.15.87:0/3760003205 learned_addr learned my addr 172.21.15.87:0/3760003205 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): get_auth_request con 0x7f47700678a0 auth_method 0 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f47754f0640 10 monclient(hunting): get_auth_request con 0x7f4770140980 auth_method 0 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f47754f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f47754f0640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f47754f0640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f476ffff640 10 monclient(hunting): get_auth_request con 0x7f47700afb00 auth_method 0 2024-07-27T15:17:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f476ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f476ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f476ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.985+0000 7f4774cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f476ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.986+0000 7f47754f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 10 monclient(hunting): handle_auth_done global_id 21922 payload 824 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 -- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4770140980 msgr2=0x7f4770136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4770140980 0x7f4770136d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 -- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47700afb00 msgr2=0x7f4770136840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47700afb00 0x7f4770136840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 10 monclient: _finish_hunting 0 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 monclient: found mon.noname-b 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.988+0000 7f4774cef640 1 -- 172.21.15.87:0/3760003205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47700ab910 con 0x7f47700678a0 2024-07-27T15:17:55.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.990+0000 7f4774cef640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 secure :-1 s=READY pgs=5031 cs=0 l=1 rev1=1 crypto rx=0x7f476400e890 tx=0x7f476400ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:55.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.990+0000 7f476d7fa640 10 client.?.objecter ms_handle_connect 0x7f47700678a0 2024-07-27T15:17:55.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.990+0000 7f476d7fa640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764017070 con 0x7f47700678a0 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: dump: 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: _finish_auth 0 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: _check_auth_tickets 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476400c940 con 0x7f47700678a0 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476400cd10 con 0x7f47700678a0 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 5 monclient: authenticate success, global_id 21922 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 10 monclient: _renew_subs 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47700abbf0 con 0x7f47700678a0 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476d7fa640 10 monclient: dump: 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f476e7fc640 4 set_mon_vals no callback set 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 10 client.21922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 10 monclient: _renew_subs 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47700aab50 con 0x7f47700678a0 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.994+0000 7f4776f53640 1 librados: init done 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.995+0000 7f4776f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.995+0000 7f4776f53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.995+0000 7f4776f53640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:55.995+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4770137320 con 0x7f47700678a0 2024-07-27T15:17:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.243+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f476401a070 con 0x7f47700678a0 2024-07-27T15:17:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.244+0000 7f476d7fa640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 0x7f475406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.244+0000 7f476ffff640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 0x7f475406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.244+0000 7f476ffff640 10 monclient: get_auth_request con 0x7f475406d7f0 auth_method 0 2024-07-27T15:17:56.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.245+0000 7f476d7fa640 10 client.21922.objecter ms_handle_connect 0x7f475406d7f0 2024-07-27T15:17:56.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.245+0000 7f476ffff640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 0x7f475406fcb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f475c036000 tx=0x7f475c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.246+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f4764057970 con 0x7f47700678a0 2024-07-27T15:17:56.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.246+0000 7f476d7fa640 10 client.21922.objecter ms_dispatch 0x7f4770000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:56.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.246+0000 7f476d7fa640 3 client.21922.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:56.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.246+0000 7f476d7fa640 3 client.21922.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:56.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.246+0000 7f476d7fa640 20 client.21922.objecter dump_active .. 0 homeless 2024-07-27T15:17:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.258+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f4770137320 con 0x7f47700678a0 2024-07-27T15:17:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.258+0000 7f476d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.258+0000 7f476d7fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:56.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.466+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f4770075c00 con 0x7f475406d7f0 2024-07-27T15:17:56.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.467+0000 7f476d7fa640 1 -- 172.21.15.87:0/3760003205 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4770075c00 con 0x7f475406d7f0 2024-07-27T15:17:56.468 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 10 librados: watch_flush enter 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 10 librados: watch_flush exit 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 20 client.21922.objecter shutdown clearing up homeless session... 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 10 client.21922.objecter successfully canceled tick 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 msgr2=0x7f475406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 0x7f475406fcb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f475c036000 tx=0x7f475c03c040 comp rx=0 tx=0).stop 2024-07-27T15:17:56.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 10 monclient: shutdown 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 msgr2=0x7f47701444c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 secure :-1 s=READY pgs=5031 cs=0 l=1 rev1=1 crypto rx=0x7f476400e890 tx=0x7f476400ed50 comp rx=0 tx=0).stop 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 shutdown_connections 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f475406d7f0 0x7f475406fcb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4770140980 0x7f4770136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47700afb00 0x7f4770136840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 --2- 172.21.15.87:0/3760003205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47700678a0 0x7f47701444c0 unknown :-1 s=CLOSED pgs=5031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.469+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 >> 172.21.15.87:0/3760003205 conn(0x7f47700b5920 msgr2=0x7f4770139d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.470+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 shutdown_connections 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.470+0000 7f4776f53640 1 -- 172.21.15.87:0/3760003205 wait complete. 2024-07-27T15:17:56.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.470+0000 7f4776f53640 1 librados: shutdown 2024-07-27T15:17:56.483 INFO:tasks.ceph:Scrubbing osd.3 2024-07-27T15:17:56.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:56.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 10 monclient: get_monmap_and_config 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 10 monclient: build_initial_monmap 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 10 monclient: monmap: 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:56.589094+0000 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:56.589094+0000 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:56.590 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 1 Processor -- start 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 1 -- start start 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.588+0000 7fc764b88640 10 monclient: init 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fc7600b56c0 0x7fc76013fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-a con 0x7fc7600b56c0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7601402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-b con 0x7fc7600b4c40 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7600b3730 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7600b42b0 con 0x7fc7600b3730 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fc7600b5aa0 con 0x7fc7600b56c0 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc7600b51f0 con 0x7fc7600b4c40 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.589+0000 7fc764b88640 10 monclient(hunting): authenticate will time out at 3416.615512s 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75ed9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc7600b3730 0x7fc7600b3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52906/0 (socket says 172.21.15.87:52906) 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75ed9d640 1 -- 172.21.15.87:0/3619085869 learned_addr learned my addr 172.21.15.87:0/3619085869 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 1 --2- 172.21.15.87:0/3619085869 >> v2:172.21.15.87:3300/0 conn(0x7fc7600b56c0 0x7fc76013fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75dd9b640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7601402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): get_auth_request con 0x7fc7600b56c0 auth_method 0 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75dd9b640 10 monclient(hunting): get_auth_request con 0x7fc7600b4c40 auth_method 0 2024-07-27T15:17:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.590+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.591+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.591+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.591+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.591+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 1 -- 172.21.15.87:0/3619085869 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288227387 0 0) 0x7fc7600b42b0 con 0x7fc7600b3730 2024-07-27T15:17:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 10 monclient(hunting): _negotiate 2024-07-27T15:17:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 10 monclient(hunting): my global_id is 21925 2024-07-27T15:17:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75d59a640 1 -- 172.21.15.87:0/3619085869 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74c003520 con 0x7fc7600b3730 2024-07-27T15:17:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 10 monclient(hunting): handle_auth_done global_id 14778 payload 274 2024-07-27T15:17:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 1 -- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 msgr2=0x7fc7601402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7601402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 1 -- 172.21.15.87:0/3619085869 >> v1:172.21.15.112:6789/0 conn(0x7fc7600b3730 legacy=0x7fc7600b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 1 monclient: found mon.noname-a 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.592+0000 7fc75e59c640 1 -- 172.21.15.87:0/3619085869 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7600aa930 con 0x7fc7600b56c0 2024-07-27T15:17:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.594+0000 7fc75e59c640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc76013fda0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7fc7480097a0 tx=0x7fc74802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248b7c9af115fe97 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 1 -- 172.21.15.87:0/3619085869 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748038070 con 0x7fc7600b56c0 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: dump: 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: _finish_auth 0 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: _check_auth_tickets 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 1 -- 172.21.15.87:0/3619085869 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc748031700 con 0x7fc7600b56c0 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 5 monclient: authenticate success, global_id 14778 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 1 -- 172.21.15.87:0/3619085869 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748031ad0 con 0x7fc7600b56c0 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 4 set_mon_vals no callback set 2024-07-27T15:17:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc75d59a640 10 monclient: dump: 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.601 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 10 monclient: shutdown 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 -- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 msgr2=0x7fc76013fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc76013fda0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7fc7480097a0 tx=0x7fc74802b7a0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 -- 172.21.15.87:0/3619085869 shutdown_connections 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7601402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 --2- 172.21.15.87:0/3619085869 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc76013fda0 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 -- 172.21.15.87:0/3619085869 >> 172.21.15.87:0/3619085869 conn(0x7fc760073130 msgr2=0x7fc760067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 -- 172.21.15.87:0/3619085869 shutdown_connections 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 1 -- 172.21.15.87:0/3619085869 wait complete. 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 10 monclient: build_initial_monmap 2024-07-27T15:17:56.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.599+0000 7fc764b88640 10 monclient: monmap: 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 librados: starting msgr at 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 librados: starting objecter 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 Processor -- start 2024-07-27T15:17:56.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 -- start start 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 librados: setting wanted keys 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 librados: calling monclient init 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient: init 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 0x7fc76013ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-b con 0x7fc7600b3730 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7600b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-c con 0x7fc7600b4c40 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc7600b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): picked mon.noname-a con 0x7fc7600b56c0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc76014d420 con 0x7fc7600b56c0 2024-07-27T15:17:56.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc7600aae00 con 0x7fc7600b3730 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc76014d5a0 con 0x7fc7600b4c40 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.600+0000 7fc764b88640 10 monclient(hunting): authenticate will time out at 3416.626885s 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc7600b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 0x7fc76013ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75dd9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7600b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 0x7fc76013ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45408/0 (socket says 172.21.15.87:45408) 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 1 -- 172.21.15.87:0/160665119 learned_addr learned my addr 172.21.15.87:0/160665119 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 10 monclient(hunting): get_auth_request con 0x7fc7600b3730 auth_method 0 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75ed9d640 10 monclient(hunting): get_auth_request con 0x7fc7600b56c0 auth_method 0 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75dd9b640 10 monclient(hunting): get_auth_request con 0x7fc7600b4c40 auth_method 0 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.601+0000 7fc75dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.602+0000 7fc75dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 10 monclient(hunting): handle_auth_done global_id 21928 payload 824 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 -- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 msgr2=0x7fc7600b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc7600b4c40 0x7fc7600b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 -- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 msgr2=0x7fc7600b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc7600b56c0 0x7fc7600b82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 monclient: found mon.noname-b 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.604+0000 7fc75e59c640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc760144560 con 0x7fc7600b3730 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.606+0000 7fc75e59c640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 0x7fc76013ec00 secure :-1 s=READY pgs=5033 cs=0 l=1 rev1=1 crypto rx=0x7fc7480098c0 tx=0x7fc7480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.606+0000 7fc756ffd640 10 client.?.objecter ms_handle_connect 0x7fc7600b3730 2024-07-27T15:17:56.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.606+0000 7fc756ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:56.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748042030 con 0x7fc7600b3730 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: dump: 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74803d670 con 0x7fc7600b3730 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 5 monclient: authenticate success, global_id 21928 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 10 monclient: _renew_subs 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc760144840 con 0x7fc7600b3730 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748038040 con 0x7fc7600b3730 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc756ffd640 10 monclient: dump: 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.614 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 10 client.21928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 10 monclient: _renew_subs 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc75cd99640 4 set_mon_vals no callback set 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76013f6e0 con 0x7fc7600b3730 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.610+0000 7fc764b88640 1 librados: init done 2024-07-27T15:17:56.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 10 client.21928.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 15 client.21928.objecter _session_command_op_assign -1 1 2024-07-27T15:17:56.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 20 client.21928.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:56.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 10 client.21928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:56.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.611+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7600b3ba0 con 0x7fc7600b3730 2024-07-27T15:17:56.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.630+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc748045070 con 0x7fc7600b3730 2024-07-27T15:17:56.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.631+0000 7fc756ffd640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc74006d7c0 0x7fc74006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.631+0000 7fc75dd9b640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc74006d7c0 0x7fc74006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.631+0000 7fc75dd9b640 10 monclient: get_auth_request con 0x7fc74006d7c0 auth_method 0 2024-07-27T15:17:56.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.631+0000 7fc756ffd640 10 client.21928.objecter ms_handle_connect 0x7fc74006d7c0 2024-07-27T15:17:56.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.631+0000 7fc75dd9b640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc74006d7c0 0x7fc74006fc80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc7601469f0 tx=0x7fc750009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7fc748079290 con 0x7fc7600b3730 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 10 client.21928.objecter ms_dispatch 0x7fc760000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 3 client.21928.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 3 client.21928.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 10 client.21928.objecter checking command 1 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7fc740075530 0x7fc7400779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter _get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 15 client.21928.objecter _session_command_op_remove -1 1 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter _get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 15 client.21928.objecter _session_command_op_assign 3 1 2024-07-27T15:17:56.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 4 2024-07-27T15:17:56.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 10 client.21928.objecter _send_command 1 2024-07-27T15:17:56.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7400780a0 con 0x7fc740075530 2024-07-27T15:17:56.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 20 client.21928.objecter dump_active .. 0 homeless 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7fc748077490 con 0x7fc7600b3730 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc756ffd640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.634+0000 7fc75ed9d640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7fc740075530 0x7fc7400779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.635+0000 7fc75ed9d640 10 monclient: get_auth_request con 0x7fc740075530 auth_method 0 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.637+0000 7fc756ffd640 10 client.21928.objecter ms_handle_connect 0x7fc740075530 2024-07-27T15:17:56.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.637+0000 7fc75ed9d640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7fc740075530 0x7fc7400779d0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.661+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== osd.3 v2:172.21.15.87:6816/1100053525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7400780a0 con 0x7fc740075530 2024-07-27T15:17:56.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.661+0000 7fc756ffd640 10 client.21928.objecter ms_dispatch 0x7fc760000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:56.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.661+0000 7fc756ffd640 10 client.21928.objecter _finish_command 1 = system:0 2024-07-27T15:17:56.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.661+0000 7fc756ffd640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 4 2024-07-27T15:17:56.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.661+0000 7fc756ffd640 15 client.21928.objecter _session_command_op_remove 3 1 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 10 client.21928.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 15 client.21928.objecter _session_command_op_assign -1 2 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 20 client.21928.objecter _get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 20 client.21928.objecter _get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 15 client.21928.objecter _session_command_op_remove -1 2 2024-07-27T15:17:56.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 20 client.21928.objecter get_session s=0x7fc740074e60 osd=3 3 2024-07-27T15:17:56.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 15 client.21928.objecter _session_command_op_assign 3 2 2024-07-27T15:17:56.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 4 2024-07-27T15:17:56.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 10 client.21928.objecter _send_command 2 2024-07-27T15:17:56.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.673+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 --> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc7601442f0 con 0x7fc740075530 2024-07-27T15:17:56.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.686+0000 7fc756ffd640 1 -- 172.21.15.87:0/160665119 <== osd.3 v2:172.21.15.87:6816/1100053525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc7601442f0 con 0x7fc740075530 2024-07-27T15:17:56.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.686+0000 7fc756ffd640 10 client.21928.objecter ms_dispatch 0x7fc760000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.686+0000 7fc756ffd640 10 client.21928.objecter _finish_command 2 = system:0 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.686+0000 7fc756ffd640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 4 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.686+0000 7fc756ffd640 15 client.21928.objecter _session_command_op_remove 3 2 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:56.688 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:56.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 10 librados: watch_flush enter 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 10 librados: watch_flush exit 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 10 client.21928.objecter close_session for osd.3 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7fc740075530 msgr2=0x7fc7400779d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.87:6816/1100053525,v1:172.21.15.87:6817/1100053525] conn(0x7fc740075530 0x7fc7400779d0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 20 client.21928.objecter put_session s=0x7fc740074e60 osd=3 1 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 20 client.21928.objecter shutdown clearing up homeless session... 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 10 client.21928.objecter successfully canceled tick 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc74006d7c0 msgr2=0x7fc74006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc74006d7c0 0x7fc74006fc80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc7601469f0 tx=0x7fc750009190 comp rx=0 tx=0).stop 2024-07-27T15:17:56.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 10 monclient: shutdown 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 msgr2=0x7fc76013ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 --2- 172.21.15.87:0/160665119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7600b3730 0x7fc76013ec00 secure :-1 s=READY pgs=5033 cs=0 l=1 rev1=1 crypto rx=0x7fc7480098c0 tx=0x7fc7480046c0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc75ed9d640 1 -- 172.21.15.87:0/160665119 reap_dead start 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 shutdown_connections 2024-07-27T15:17:56.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 >> 172.21.15.87:0/160665119 conn(0x7fc760073130 msgr2=0x7fc76006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:56.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 shutdown_connections 2024-07-27T15:17:56.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 -- 172.21.15.87:0/160665119 wait complete. 2024-07-27T15:17:56.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.687+0000 7fc764b88640 1 librados: shutdown 2024-07-27T15:17:56.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-07-27T15:17:56.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: get_monmap_and_config 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: build_initial_monmap 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: monmap: 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:56.797554+0000 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:56.797554+0000 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.799 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 Processor -- start 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 -- start start 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: init 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f69180b1be0 0x7f69180aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): picked mon.noname-c con 0x7f69180b1be0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:56.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f69180737f0 0x7f69180b0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): picked mon.noname-b con 0x7f69180737f0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f691813fd20 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6918073340 con 0x7f691813fd20 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f69180abd80 con 0x7f69180b1be0 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f6918073c10 con 0x7f69180737f0 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.797+0000 7f692000b640 10 monclient(hunting): authenticate will time out at 3416.823860s 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691dda7640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f69180b1be0 0x7f69180aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691d5a6640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f69180737f0 0x7f69180b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691e5a8640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f691813fd20 0x7f6918145ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:58614/0 (socket says 172.21.15.87:58614) 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691e5a8640 1 -- 172.21.15.87:0/4081808669 learned_addr learned my addr 172.21.15.87:0/4081808669 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691dda7640 10 monclient(hunting): get_auth_request con 0x7f69180b1be0 auth_method 0 2024-07-27T15:17:56.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691dda7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691dda7640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691dda7640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691d5a6640 10 monclient(hunting): get_auth_request con 0x7f69180737f0 auth_method 0 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691d5a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691d5a6640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.798+0000 7f691d5a6640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.799+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 10 monclient(hunting): handle_auth_done global_id 21931 payload 274 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 1 -- 172.21.15.87:0/4081808669 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f69180737f0 msgr2=0x7f69180b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 1 --2- 172.21.15.87:0/4081808669 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f69180737f0 0x7f69180b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 1 -- 172.21.15.87:0/4081808669 >> v1:172.21.15.87:6789/0 conn(0x7f691813fd20 legacy=0x7f6918145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 10 monclient: _finish_hunting 0 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 1 monclient: found mon.noname-c 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.800+0000 7f691dda7640 1 -- 172.21.15.87:0/4081808669 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69180b0650 con 0x7f69180b1be0 2024-07-27T15:17:56.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.802+0000 7f691dda7640 1 --2- 172.21.15.87:0/4081808669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69180aa890 secure :-1 s=READY pgs=5034 cs=0 l=1 rev1=1 crypto rx=0x7f6908009dc0 tx=0x7f690802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c17f0c11f1f5d60 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 1 -- 172.21.15.87:0/4081808669 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908039070 con 0x7f69180b1be0 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 10 monclient: dump: 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 10 monclient: _finish_auth 0 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 10 monclient: _check_auth_tickets 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f691cda5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f692000b640 5 monclient: authenticate success, global_id 21931 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.804+0000 7f692000b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.805+0000 7f691cda5640 1 -- 172.21.15.87:0/4081808669 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f690802cbb0 con 0x7f69180b1be0 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f691cda5640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 4 set_mon_vals no callback set 2024-07-27T15:17:56.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f691cda5640 1 -- 172.21.15.87:0/4081808669 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908034440 con 0x7f69180b1be0 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f691cda5640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f691cda5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f691cda5640 10 monclient: dump: 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 10 monclient: shutdown 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 -- 172.21.15.87:0/4081808669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 msgr2=0x7f69180aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 --2- 172.21.15.87:0/4081808669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69180aa890 secure :-1 s=READY pgs=5034 cs=0 l=1 rev1=1 crypto rx=0x7f6908009dc0 tx=0x7f690802be00 comp rx=0 tx=0).stop 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 -- 172.21.15.87:0/4081808669 shutdown_connections 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 --2- 172.21.15.87:0/4081808669 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f69180737f0 0x7f69180b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 --2- 172.21.15.87:0/4081808669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69180aa890 unknown :-1 s=CLOSED pgs=5034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 -- 172.21.15.87:0/4081808669 >> 172.21.15.87:0/4081808669 conn(0x7f69180b4d30 msgr2=0x7f69180b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 -- 172.21.15.87:0/4081808669 shutdown_connections 2024-07-27T15:17:56.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 -- 172.21.15.87:0/4081808669 wait complete. 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 10 monclient: build_initial_monmap 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 10 monclient: monmap: 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:56.815 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 librados: starting msgr at 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.807+0000 7f692000b640 1 librados: starting objecter 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 Processor -- start 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 -- start start 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 librados: setting wanted keys 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 librados: calling monclient init 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient: init 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69180737f0 0x7f69181ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): picked mon.noname-a con 0x7f69180737f0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): picked mon.noname-b con 0x7f69180b1be0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 0x7f69181408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): picked mon.noname-c con 0x7f691813fd20 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f69181401d0 con 0x7f69180737f0 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f69180aee30 con 0x7f69180b1be0 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f6918140350 con 0x7f691813fd20 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.808+0000 7f692000b640 10 monclient(hunting): authenticate will time out at 3416.834728s 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691dda7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69180737f0 0x7f69181ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 0x7f69181408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45438/0 (socket says 172.21.15.87:45438) 2024-07-27T15:17:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 0x7f69181408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37178/0 (socket says 172.21.15.87:37178) 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 1 -- 172.21.15.87:0/3940560773 learned_addr learned my addr 172.21.15.87:0/3940560773 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691dda7640 10 monclient(hunting): get_auth_request con 0x7f69180737f0 auth_method 0 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691dda7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691dda7640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691dda7640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 10 monclient(hunting): get_auth_request con 0x7f69180b1be0 auth_method 0 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691d5a6640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 10 monclient(hunting): get_auth_request con 0x7f691813fd20 auth_method 0 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.809+0000 7f691e5a8640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:56.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691d5a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691e5a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691e5a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691e5a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.810+0000 7f691dda7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 10 monclient(hunting): handle_auth_done global_id 21934 payload 824 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 -- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 msgr2=0x7f69181408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 0x7f69181408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 -- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69180737f0 msgr2=0x7f69181ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69180737f0 0x7f69181ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:56.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 10 monclient: _finish_hunting 0 2024-07-27T15:17:56.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 monclient: found mon.noname-b 2024-07-27T15:17:56.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:56.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.812+0000 7f691d5a6640 1 -- 172.21.15.87:0/3940560773 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6918143920 con 0x7f69180b1be0 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.814+0000 7f691d5a6640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 secure :-1 s=READY pgs=5035 cs=0 l=1 rev1=1 crypto rx=0x7f690c00e980 tx=0x7f690c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.814+0000 7f69167fc640 10 client.?.objecter ms_handle_connect 0x7f69180b1be0 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.814+0000 7f69167fc640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.818+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c017070 con 0x7f69180b1be0 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: dump: 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: _finish_auth 0 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: _check_auth_tickets 2024-07-27T15:17:56.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f690c00ca00 con 0x7f69180b1be0 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 5 monclient: authenticate success, global_id 21934 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _renew_subs 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6918144730 con 0x7f69180b1be0 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c00cdd0 con 0x7f69180b1be0 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f69167fc640 10 monclient: dump: 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.823 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f6917fff640 4 set_mon_vals no callback set 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 client.21934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _renew_subs 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6918140e30 con 0x7f69180b1be0 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 1 librados: init done 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:56.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:56.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.819+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69180b0110 con 0x7f69180b1be0 2024-07-27T15:17:56.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f690c01a070 con 0x7f69180b1be0 2024-07-27T15:17:56.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f69167fc640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 0x7f690406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:56.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f691dda7640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 0x7f690406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:56.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f691dda7640 10 monclient: get_auth_request con 0x7f690406d7c0 auth_method 0 2024-07-27T15:17:56.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f691dda7640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 0x7f690406fc80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6908036000 tx=0x7f6908030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:56.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.838+0000 7f69167fc640 10 client.21934.objecter ms_handle_connect 0x7f690406d7c0 2024-07-27T15:17:56.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.840+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f690c057a10 con 0x7f69180b1be0 2024-07-27T15:17:56.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.840+0000 7f69167fc640 10 client.21934.objecter ms_dispatch 0x7f6918000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:56.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.840+0000 7f69167fc640 3 client.21934.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:56.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.840+0000 7f69167fc640 3 client.21934.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:56.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:56.840+0000 7f69167fc640 20 client.21934.objecter dump_active .. 0 homeless 2024-07-27T15:17:57.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.083+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f690c059970 con 0x7f69180b1be0 2024-07-27T15:17:57.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.083+0000 7f69167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:57.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.083+0000 7f69167fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:57.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.272+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f6918073f00 con 0x7f690406d7c0 2024-07-27T15:17:57.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.273+0000 7f69167fc640 1 -- 172.21.15.87:0/3940560773 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6918073f00 con 0x7f690406d7c0 2024-07-27T15:17:57.273 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 3 to deep-scrub 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.274+0000 7f692000b640 10 librados: watch_flush enter 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.274+0000 7f692000b640 10 librados: watch_flush exit 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.274+0000 7f692000b640 20 client.21934.objecter shutdown clearing up homeless session... 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.274+0000 7f692000b640 10 client.21934.objecter successfully canceled tick 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 msgr2=0x7f690406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 0x7f690406fc80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6908036000 tx=0x7f6908030000 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 10 monclient: shutdown 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 msgr2=0x7f69181433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 secure :-1 s=READY pgs=5035 cs=0 l=1 rev1=1 crypto rx=0x7f690c00e980 tx=0x7f690c00ee40 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 shutdown_connections 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f690406d7c0 0x7f690406fc80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f691813fd20 0x7f69181408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69180b1be0 0x7f69181433e0 unknown :-1 s=CLOSED pgs=5035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 --2- 172.21.15.87:0/3940560773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69180737f0 0x7f69181ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 >> 172.21.15.87:0/3940560773 conn(0x7f69180b4d30 msgr2=0x7f69180b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 shutdown_connections 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 -- 172.21.15.87:0/3940560773 wait complete. 2024-07-27T15:17:57.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.275+0000 7f692000b640 1 librados: shutdown 2024-07-27T15:17:57.286 INFO:tasks.ceph:Scrubbing osd.4 2024-07-27T15:17:57.286 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:57.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 10 monclient: get_monmap_and_config 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 10 monclient: build_initial_monmap 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 10 monclient: monmap: 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:57.390135+0000 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:57.390135+0000 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:57.392 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 1 Processor -- start 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 1 -- start start 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.389+0000 7f3fb494b640 10 monclient: init 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3fb00b56c0 0x7f3fb013fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-c con 0x7f3fb00b56c0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f3fb00b4c40 0x7f3fb01402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-a con 0x7f3fb00b4c40 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3fb00b3730 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fb00aa930 con 0x7f3fb00b3730 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f3fb00b5aa0 con 0x7f3fb00b4c40 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f3fb00b51f0 con 0x7f3fb00b56c0 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.390+0000 7f3fb494b640 10 monclient(hunting): authenticate will time out at 3417.416568s 2024-07-27T15:17:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3fae59c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3fb00b56c0 0x7f3fb013fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3faed9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3fb00b3730 0x7f3fb00b3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52916/0 (socket says 172.21.15.87:52916) 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3faed9d640 1 -- 172.21.15.87:0/2597510661 learned_addr learned my addr 172.21.15.87:0/2597510661 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3fae59c640 10 monclient(hunting): get_auth_request con 0x7f3fb00b56c0 auth_method 0 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3fae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3fae59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.391+0000 7f3fae59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.392+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.392+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.392+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.393+0000 7f3fadd9b640 1 --2- 172.21.15.87:0/2597510661 >> v2:172.21.15.87:3300/0 conn(0x7f3fb00b4c40 0x7f3fb01402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.393+0000 7f3fadd9b640 10 monclient(hunting): get_auth_request con 0x7f3fb00b4c40 auth_method 0 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.393+0000 7f3fadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.393+0000 7f3fadd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.393+0000 7f3fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 10 monclient(hunting): handle_auth_done global_id 21937 payload 274 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 1 -- 172.21.15.87:0/2597510661 >> v2:172.21.15.87:3300/0 conn(0x7f3fb00b4c40 msgr2=0x7f3fb01402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 1 --2- 172.21.15.87:0/2597510661 >> v2:172.21.15.87:3300/0 conn(0x7f3fb00b4c40 0x7f3fb01402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 1 -- 172.21.15.87:0/2597510661 >> v1:172.21.15.112:6789/0 conn(0x7f3fb00b3730 legacy=0x7f3fb00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 1 monclient: found mon.noname-c 2024-07-27T15:17:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.394+0000 7f3fae59c640 1 -- 172.21.15.87:0/2597510661 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb00b4290 con 0x7f3fb00b56c0 2024-07-27T15:17:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.398+0000 7f3fae59c640 1 --2- 172.21.15.87:0/2597510661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b56c0 0x7f3fb013fda0 secure :-1 s=READY pgs=5037 cs=0 l=1 rev1=1 crypto rx=0x7f3fb00b9a60 tx=0x7f3fa802bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33475751d44eeb9a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 1 -- 172.21.15.87:0/2597510661 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8039070 con 0x7f3fb00b56c0 2024-07-27T15:17:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: dump: 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: _finish_auth 0 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: _check_auth_tickets 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 1 -- 172.21.15.87:0/2597510661 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fa802cb60 con 0x7f3fb00b56c0 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fb494b640 5 monclient: authenticate success, global_id 21937 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fb494b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fb494b640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 1 -- 172.21.15.87:0/2597510661 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8034440 con 0x7f3fb00b56c0 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fb494b640 4 set_mon_vals no callback set 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.402+0000 7f3fad59a640 10 monclient: dump: 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.406 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.407 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 10 monclient: shutdown 2024-07-27T15:17:57.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 -- 172.21.15.87:0/2597510661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b56c0 msgr2=0x7f3fb013fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 --2- 172.21.15.87:0/2597510661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b56c0 0x7f3fb013fda0 secure :-1 s=READY pgs=5037 cs=0 l=1 rev1=1 crypto rx=0x7f3fb00b9a60 tx=0x7f3fa802bdb0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 -- 172.21.15.87:0/2597510661 shutdown_connections 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 --2- 172.21.15.87:0/2597510661 >> v2:172.21.15.87:3300/0 conn(0x7f3fb00b4c40 0x7f3fb01402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 --2- 172.21.15.87:0/2597510661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b56c0 0x7f3fb013fda0 unknown :-1 s=CLOSED pgs=5037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 -- 172.21.15.87:0/2597510661 >> 172.21.15.87:0/2597510661 conn(0x7f3fb0073130 msgr2=0x7f3fb0067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 -- 172.21.15.87:0/2597510661 shutdown_connections 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.403+0000 7f3fb494b640 1 -- 172.21.15.87:0/2597510661 wait complete. 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 10 monclient: build_initial_monmap 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 10 monclient: monmap: 2024-07-27T15:17:57.412 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:57.413 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:57.413 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:57.413 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 librados: starting msgr at 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 librados: starting objecter 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 Processor -- start 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 -- start start 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 librados: setting wanted keys 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 1 librados: calling monclient init 2024-07-27T15:17:57.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.404+0000 7f3fb494b640 10 monclient: init 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3fb00b3730 0x7f3fb00ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-c con 0x7f3fb00b3730 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 0x7f3fb0149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-b con 0x7f3fb00b4c40 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fb00b56c0 0x7f3fb0149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): picked mon.noname-a con 0x7f3fb00b56c0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb014d420 con 0x7f3fb00b56c0 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb00aae00 con 0x7f3fb00b4c40 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb014d5a0 con 0x7f3fb00b3730 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fb494b640 10 monclient(hunting): authenticate will time out at 3417.431327s 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fadd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 0x7f3fb0149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.405+0000 7f3fae59c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3fb00b3730 0x7f3fb00ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fb00b56c0 0x7f3fb0149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 0x7f3fb0149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45460/0 (socket says 172.21.15.87:45460) 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 1 -- 172.21.15.87:0/1332946266 learned_addr learned my addr 172.21.15.87:0/1332946266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): get_auth_request con 0x7f3fb00b4c40 auth_method 0 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): get_auth_request con 0x7f3fb00b56c0 auth_method 0 2024-07-27T15:17:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): get_auth_request con 0x7f3fb00b3730 auth_method 0 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3faed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.406+0000 7f3fae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 10 monclient(hunting): handle_auth_done global_id 21943 payload 824 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 -- 172.21.15.87:0/1332946266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3fb00b3730 msgr2=0x7f3fb00ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3fb00b3730 0x7f3fb00ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 -- 172.21.15.87:0/1332946266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fb00b56c0 msgr2=0x7f3fb0149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fb00b56c0 0x7f3fb0149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 monclient: found mon.noname-b 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.409+0000 7f3fadd9b640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb00ada20 con 0x7f3fb00b4c40 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.411+0000 7f3fadd9b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 0x7f3fb0149750 secure :-1 s=READY pgs=5038 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800e9b0 tx=0x7f3f9800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.411+0000 7f3fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f3fb00b4c40 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.411+0000 7f3fa6ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f98017070 con 0x7f3fb00b4c40 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: dump: 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f9800c9a0 con 0x7f3fb00b4c40 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9800cd90 con 0x7f3fb00b4c40 2024-07-27T15:17:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fa6ffd640 10 monclient: dump: 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fb494b640 5 monclient: authenticate success, global_id 21943 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fb494b640 10 monclient: _renew_subs 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.414+0000 7f3fb494b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb00add60 con 0x7f3fb00b4c40 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3facd99640 4 set_mon_vals no callback set 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 client.21943.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 monclient: _renew_subs 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb00aca60 con 0x7f3fb00b4c40 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 1 librados: init done 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 client.21943.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 15 client.21943.objecter _session_command_op_assign -1 1 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 20 client.21943.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 client.21943.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.415+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3fb00acfe0 con 0x7f3fb00b4c40 2024-07-27T15:17:57.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.434+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3f9801a070 con 0x7f3fb00b4c40 2024-07-27T15:17:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.435+0000 7f3fa6ffd640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f9006d7c0 0x7f3f9006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.435+0000 7f3fae59c640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f9006d7c0 0x7f3f9006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.435+0000 7f3fae59c640 10 monclient: get_auth_request con 0x7f3f9006d7c0 auth_method 0 2024-07-27T15:17:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.435+0000 7f3fae59c640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f9006d7c0 0x7f3f9006fc80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8036000 tx=0x7f3fa8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.435+0000 7f3fa6ffd640 10 client.21943.objecter ms_handle_connect 0x7f3f9006d7c0 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f3f980575e0 con 0x7f3fb00b4c40 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 10 client.21943.objecter ms_dispatch 0x7f3fb0000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 3 client.21943.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 3 client.21943.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 10 client.21943.objecter checking command 1 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f3f90075530 0x7f3f900779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter _get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 15 client.21943.objecter _session_command_op_remove -1 1 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter _get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 15 client.21943.objecter _session_command_op_assign 4 1 2024-07-27T15:17:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 4 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 10 client.21943.objecter _send_command 1 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f98007e50 con 0x7f3f90075530 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.438+0000 7f3fa6ffd640 20 client.21943.objecter dump_active .. 0 homeless 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.439+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7f3f980557e0 con 0x7f3fb00b4c40 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.439+0000 7f3fa6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.439+0000 7f3faed9d640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f3f90075530 0x7f3f900779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.439+0000 7f3faed9d640 10 monclient: get_auth_request con 0x7f3f90075530 auth_method 0 2024-07-27T15:17:57.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.442+0000 7f3faed9d640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f3f90075530 0x7f3f900779d0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.442+0000 7f3fa6ffd640 10 client.21943.objecter ms_handle_connect 0x7f3f90075530 2024-07-27T15:17:57.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.465+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== osd.4 v2:172.21.15.112:6826/1578415927 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f98007e50 con 0x7f3f90075530 2024-07-27T15:17:57.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.465+0000 7f3fa6ffd640 10 client.21943.objecter ms_dispatch 0x7f3fb0000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:57.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.465+0000 7f3fa6ffd640 10 client.21943.objecter _finish_command 1 = system:0 2024-07-27T15:17:57.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.465+0000 7f3fa6ffd640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 4 2024-07-27T15:17:57.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.465+0000 7f3fa6ffd640 15 client.21943.objecter _session_command_op_remove 4 1 2024-07-27T15:17:57.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.477+0000 7f3fb494b640 10 client.21943.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:57.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 15 client.21943.objecter _session_command_op_assign -1 2 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 20 client.21943.objecter _get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 20 client.21943.objecter _get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 15 client.21943.objecter _session_command_op_remove -1 2 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 20 client.21943.objecter get_session s=0x7f3f90074e60 osd=4 3 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 15 client.21943.objecter _session_command_op_assign 4 2 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 4 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 10 client.21943.objecter _send_command 2 2024-07-27T15:17:57.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.478+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 --> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3fb014a420 con 0x7f3f90075530 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.493+0000 7f3fa6ffd640 1 -- 172.21.15.87:0/1332946266 <== osd.4 v2:172.21.15.112:6826/1578415927 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3fb014a420 con 0x7f3f90075530 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.493+0000 7f3fa6ffd640 10 client.21943.objecter ms_dispatch 0x7f3fb0000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.493+0000 7f3fa6ffd640 10 client.21943.objecter _finish_command 2 = system:0 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.493+0000 7f3fa6ffd640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 4 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.493+0000 7f3fa6ffd640 15 client.21943.objecter _session_command_op_remove 4 2 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:57.495 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:57.496 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 10 librados: watch_flush enter 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 10 librados: watch_flush exit 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 10 client.21943.objecter close_session for osd.4 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f3f90075530 msgr2=0x7f3f900779d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6826/1578415927,v1:172.21.15.112:6827/1578415927] conn(0x7f3f90075530 0x7f3f900779d0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 20 client.21943.objecter put_session s=0x7f3f90074e60 osd=4 1 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 20 client.21943.objecter shutdown clearing up homeless session... 2024-07-27T15:17:57.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 10 client.21943.objecter successfully canceled tick 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f9006d7c0 msgr2=0x7f3f9006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3f9006d7c0 0x7f3f9006fc80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8036000 tx=0x7f3fa8030000 comp rx=0 tx=0).stop 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 10 monclient: shutdown 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 msgr2=0x7f3fb0149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 --2- 172.21.15.87:0/1332946266 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fb00b4c40 0x7f3fb0149750 secure :-1 s=READY pgs=5038 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800e9b0 tx=0x7f3f9800ee70 comp rx=0 tx=0).stop 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3faed9d640 1 -- 172.21.15.87:0/1332946266 reap_dead start 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 shutdown_connections 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 >> 172.21.15.87:0/1332946266 conn(0x7f3fb0073130 msgr2=0x7f3fb00b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 shutdown_connections 2024-07-27T15:17:57.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 -- 172.21.15.87:0/1332946266 wait complete. 2024-07-27T15:17:57.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.494+0000 7f3fb494b640 1 librados: shutdown 2024-07-27T15:17:57.507 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.608+0000 7f6c12ccc640 10 monclient: get_monmap_and_config 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.608+0000 7f6c12ccc640 10 monclient: build_initial_monmap 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.608+0000 7f6c12ccc640 10 monclient: monmap: 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:57.609921+0000 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:57.609921+0000 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:57.615 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.609+0000 7f6c12ccc640 1 Processor -- start 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.609+0000 7f6c12ccc640 1 -- start start 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.609+0000 7f6c12ccc640 10 monclient: init 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6c0c0aeae0 0x7f6c0c0b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c0aeae0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c0c13fb10 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c0c146140 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c0c070360 con 0x7f6c0c13fb10 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c0c0b12b0 con 0x7f6c0c146140 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f6c0c0aeec0 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.610+0000 7f6c12ccc640 10 monclient(hunting): authenticate will time out at 3417.636491s 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c10a68640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6c0c0aeae0 0x7f6c0c0b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c11269640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6c0c146140 0x7f6c0c1465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52930/0 (socket says 172.21.15.87:52930) 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c11269640 1 -- 172.21.15.87:0/1302203114 learned_addr learned my addr 172.21.15.87:0/1302203114 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c10a68640 10 monclient(hunting): get_auth_request con 0x7f6c0c0aeae0 auth_method 0 2024-07-27T15:17:57.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c10a68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c10a68640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.611+0000 7f6c10a68640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.612+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.612+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.612+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 1 -- 172.21.15.87:0/1302203114 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480386124 0 0) 0x7f6c0c0b12b0 con 0x7f6c0c146140 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 10 monclient(hunting): _negotiate 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 10 monclient(hunting): my global_id is 21946 2024-07-27T15:17:57.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.613+0000 7f6c0b7fe640 1 -- 172.21.15.87:0/1302203114 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bf4003520 con 0x7f6c0c146140 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 10 monclient(hunting): handle_auth_done global_id 14796 payload 274 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 1 -- 172.21.15.87:0/1302203114 >> v1:172.21.15.112:6789/0 conn(0x7f6c0c146140 legacy=0x7f6c0c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 1 -- 172.21.15.87:0/1302203114 >> v1:172.21.15.87:6789/0 conn(0x7f6c0c13fb10 legacy=0x7f6c0c13ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 10 monclient: _finish_hunting 0 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 1 monclient: found mon.noname-a 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:57.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.616+0000 7f6c10a68640 1 -- 172.21.15.87:0/1302203114 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c140700 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.619+0000 7f6c10a68640 1 --2- 172.21.15.87:0/1302203114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c0b07b0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097a0 tx=0x7f6bfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b5cbff23f43172 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 1 -- 172.21.15.87:0/1302203114 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc035020 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: dump: 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.626 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: _finish_auth 0 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: _check_auth_tickets 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 1 -- 172.21.15.87:0/1302203114 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bfc031830 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c12ccc640 5 monclient: authenticate success, global_id 14796 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c12ccc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 1 -- 172.21.15.87:0/1302203114 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc031c00 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c12ccc640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c12ccc640 4 set_mon_vals no callback set 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:57.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.624+0000 7f6c0b7fe640 10 monclient: dump: 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.628 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 10 monclient: shutdown 2024-07-27T15:17:57.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:57.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 -- 172.21.15.87:0/1302203114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 msgr2=0x7f6c0c0b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/1302203114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c0b07b0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097a0 tx=0x7f6bfc02b8d0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 -- 172.21.15.87:0/1302203114 shutdown_connections 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/1302203114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c0b07b0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 -- 172.21.15.87:0/1302203114 >> 172.21.15.87:0/1302203114 conn(0x7f6c0c068050 msgr2=0x7f6c0c0b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 -- 172.21.15.87:0/1302203114 shutdown_connections 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.625+0000 7f6c12ccc640 1 -- 172.21.15.87:0/1302203114 wait complete. 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 10 monclient: build_initial_monmap 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 10 monclient: monmap: 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:57.632 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 librados: starting msgr at 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 librados: starting objecter 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 Processor -- start 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 -- start start 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 librados: setting wanted keys 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 1 librados: calling monclient init 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.626+0000 7f6c12ccc640 10 monclient: init 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T15:17:57.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c13c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c0aeae0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0c13fb10 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c0c146140 0x7f6c0c142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): picked mon.noname-c con 0x7f6c0c146140 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c14e6f0 con 0x7f6c0c0aeae0 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c0b0e20 con 0x7f6c0c13fb10 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c14e870 con 0x7f6c0c146140 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.627+0000 7f6c12ccc640 10 monclient(hunting): authenticate will time out at 3417.653616s 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c13c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c0c146140 0x7f6c0c142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45474/0 (socket says 172.21.15.87:45474) 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 1 -- 172.21.15.87:0/2589193212 learned_addr learned my addr 172.21.15.87:0/2589193212 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): get_auth_request con 0x7f6c0c13fb10 auth_method 0 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): get_auth_request con 0x7f6c0c0aeae0 auth_method 0 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): get_auth_request con 0x7f6c0c146140 auth_method 0 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c11269640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:57.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.628+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:57.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.629+0000 7f6c10a68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:57.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 10 monclient(hunting): handle_auth_done global_id 21949 payload 824 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 -- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c0c146140 msgr2=0x7f6c0c142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c0c146140 0x7f6c0c142110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 -- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 msgr2=0x7f6c0c13c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c13c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 10 monclient: _finish_hunting 0 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 monclient: found mon.noname-b 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.631+0000 7f6c0bfff640 1 -- 172.21.15.87:0/2589193212 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c143780 con 0x7f6c0c13fb10 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.633+0000 7f6c0bfff640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 secure :-1 s=READY pgs=5040 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c400 tx=0x7f6c0000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.633+0000 7f6c097fa640 10 client.?.objecter ms_handle_connect 0x7f6c0c13fb10 2024-07-27T15:17:57.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.634+0000 7f6c097fa640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c000052d0 con 0x7f6c0c13fb10 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 10 monclient: dump: 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.640 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 10 monclient: _finish_auth 0 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 10 monclient: _check_auth_tickets 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.635+0000 7f6c097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 5 monclient: authenticate success, global_id 21949 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 monclient: _renew_subs 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c141bb0 con 0x7f6c0c13fb10 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 client.21949.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 monclient: _renew_subs 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c142830 con 0x7f6c0c13fb10 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 1 librados: init done 2024-07-27T15:17:57.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.636+0000 7f6c12ccc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c12ccc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c12ccc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c0b07b0 con 0x7f6c0c13fb10 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c00005470 con 0x7f6c0c13fb10 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00020060 con 0x7f6c0c13fb10 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c097fa640 10 monclient: dump: 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:57.642 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:57.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.637+0000 7f6c0affd640 4 set_mon_vals no callback set 2024-07-27T15:17:57.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.885+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c00007220 con 0x7f6c0c13fb10 2024-07-27T15:17:57.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.886+0000 7f6c097fa640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 0x7f6bf806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:57.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.886+0000 7f6c10a68640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 0x7f6bf806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:57.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.886+0000 7f6c10a68640 10 monclient: get_auth_request con 0x7f6bf806d7c0 auth_method 0 2024-07-27T15:17:57.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.887+0000 7f6c10a68640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 0x7f6bf806fc80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc036000 tx=0x7f6bfc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:57.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.887+0000 7f6c097fa640 10 client.21949.objecter ms_handle_connect 0x7f6bf806d7c0 2024-07-27T15:17:57.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.888+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f6c00056c30 con 0x7f6c0c13fb10 2024-07-27T15:17:57.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.888+0000 7f6c097fa640 10 client.21949.objecter ms_dispatch 0x7f6c0c000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:57.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.888+0000 7f6c097fa640 3 client.21949.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:57.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.888+0000 7f6c097fa640 3 client.21949.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:57.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.888+0000 7f6c097fa640 20 client.21949.objecter dump_active .. 0 homeless 2024-07-27T15:17:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.898+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f6c00058b90 con 0x7f6c0c13fb10 2024-07-27T15:17:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.898+0000 7f6c097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:57.898+0000 7f6c097fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:58.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.081+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f6c0c0b4410 con 0x7f6bf806d7c0 2024-07-27T15:17:58.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.082+0000 7f6c097fa640 1 -- 172.21.15.87:0/2589193212 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6c0c0b4410 con 0x7f6bf806d7c0 2024-07-27T15:17:58.083 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 4 to deep-scrub 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 10 librados: watch_flush enter 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 10 librados: watch_flush exit 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 20 client.21949.objecter shutdown clearing up homeless session... 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 10 client.21949.objecter successfully canceled tick 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 msgr2=0x7f6bf806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 0x7f6bf806fc80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc036000 tx=0x7f6bfc008030 comp rx=0 tx=0).stop 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 10 monclient: shutdown 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 msgr2=0x7f6c0c141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 secure :-1 s=READY pgs=5040 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c400 tx=0x7f6c0000c8c0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 shutdown_connections 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f6bf806d7c0 0x7f6bf806fc80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c0c146140 0x7f6c0c142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c13fb10 0x7f6c0c141670 unknown :-1 s=CLOSED pgs=5040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 --2- 172.21.15.87:0/2589193212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c0c0aeae0 0x7f6c0c13c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 >> 172.21.15.87:0/2589193212 conn(0x7f6c0c068050 msgr2=0x7f6c0c0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 shutdown_connections 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.084+0000 7f6c12ccc640 1 -- 172.21.15.87:0/2589193212 wait complete. 2024-07-27T15:17:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.085+0000 7f6c12ccc640 1 librados: shutdown 2024-07-27T15:17:58.097 INFO:tasks.ceph:Scrubbing osd.5 2024-07-27T15:17:58.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.211+0000 7f2fad851640 10 monclient: get_monmap_and_config 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.211+0000 7f2fad851640 10 monclient: build_initial_monmap 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.211+0000 7f2fad851640 10 monclient: monmap: 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:58.212329+0000 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:58.212329+0000 2024-07-27T15:17:58.212 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:58.213 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 Processor -- start 2024-07-27T15:17:58.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 -- start start 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient: init 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f2fa80b56c0 0x7f2fa813fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa80b56c0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2fa80b4c40 0x7f2fa81402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa80b4c40 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fa80b3730 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa80aa930 con 0x7f2fa80b3730 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa80b5aa0 con 0x7f2fa80b56c0 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa80b51f0 con 0x7f2fa80b4c40 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.212+0000 7f2fad851640 10 monclient(hunting): authenticate will time out at 3418.238758s 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.213+0000 7f2fa67fc640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2fa80b4c40 0x7f2fa81402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.213+0000 7f2fa77fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f2fa80b3730 0x7f2fa80b3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52938/0 (socket says 172.21.15.87:52938) 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.213+0000 7f2fa77fe640 1 -- 172.21.15.87:0/3941852949 learned_addr learned my addr 172.21.15.87:0/3941852949 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): get_auth_request con 0x7f2fa80b4c40 auth_method 0 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.214+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.216+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_done global_id 21952 payload 274 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.216+0000 7f2fa67fc640 1 -- 172.21.15.87:0/3941852949 >> v2:172.21.15.87:3300/0 conn(0x7f2fa80b56c0 msgr2=0x7f2fa813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.216+0000 7f2fa67fc640 1 --2- 172.21.15.87:0/3941852949 >> v2:172.21.15.87:3300/0 conn(0x7f2fa80b56c0 0x7f2fa813fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 1 -- 172.21.15.87:0/3941852949 >> v1:172.21.15.112:6789/0 conn(0x7f2fa80b3730 legacy=0x7f2fa80b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 10 monclient: _finish_hunting 0 2024-07-27T15:17:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 1 monclient: found mon.noname-c 2024-07-27T15:17:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.217+0000 7f2fa67fc640 1 -- 172.21.15.87:0/3941852949 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa80b4290 con 0x7f2fa80b4c40 2024-07-27T15:17:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.221+0000 7f2fa67fc640 1 --2- 172.21.15.87:0/3941852949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b4c40 0x7f2fa81402e0 secure :-1 s=READY pgs=5042 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80ab770 tx=0x7f2f9c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f9b97c621ba4f1 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 1 -- 172.21.15.87:0/3941852949 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c038070 con 0x7f2fa80b4c40 2024-07-27T15:17:58.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 10 monclient: dump: 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.227 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 10 monclient: _finish_auth 0 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 10 monclient: _check_auth_tickets 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fad851640 5 monclient: authenticate success, global_id 21952 2024-07-27T15:17:58.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.225+0000 7f2fad851640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 1 -- 172.21.15.87:0/3941852949 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c031710 con 0x7f2fa80b4c40 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 1 -- 172.21.15.87:0/3941852949 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c031ae0 con 0x7f2fa80b4c40 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fa5ffb640 10 monclient: dump: 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fad851640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:58.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.278+0000 7f2fad851640 4 set_mon_vals no callback set 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 10 monclient: shutdown 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 -- 172.21.15.87:0/3941852949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b4c40 msgr2=0x7f2fa81402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 --2- 172.21.15.87:0/3941852949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b4c40 0x7f2fa81402e0 secure :-1 s=READY pgs=5042 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80ab770 tx=0x7f2f9c02b7b0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 -- 172.21.15.87:0/3941852949 shutdown_connections 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 --2- 172.21.15.87:0/3941852949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b4c40 0x7f2fa81402e0 unknown :-1 s=CLOSED pgs=5042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 --2- 172.21.15.87:0/3941852949 >> v2:172.21.15.87:3300/0 conn(0x7f2fa80b56c0 0x7f2fa813fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 -- 172.21.15.87:0/3941852949 >> 172.21.15.87:0/3941852949 conn(0x7f2fa8073130 msgr2=0x7f2fa8067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 -- 172.21.15.87:0/3941852949 shutdown_connections 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 1 -- 172.21.15.87:0/3941852949 wait complete. 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 10 monclient: build_initial_monmap 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.279+0000 7f2fad851640 10 monclient: monmap: 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 librados: starting msgr at 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 librados: starting objecter 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 Processor -- start 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 -- start start 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 librados: setting wanted keys 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 librados: calling monclient init 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 10 monclient: init 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fa80b3730 0x7f2fa8144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.280+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa80b3730 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 0x7f2fa8132b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa80b4c40 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b56c0 0x7f2fa81330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa80b56c0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa814d420 con 0x7f2fa80b3730 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80aae00 con 0x7f2fa80b56c0 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa814d5a0 con 0x7f2fa80b4c40 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fad851640 10 monclient(hunting): authenticate will time out at 3418.307059s 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fa67fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 0x7f2fa8132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fa6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fa80b3730 0x7f2fa8144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.281+0000 7f2fa67fc640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 0x7f2fa8132b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37216/0 (socket says 172.21.15.87:37216) 2024-07-27T15:17:58.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 1 -- 172.21.15.87:0/2808416656 learned_addr learned my addr 172.21.15.87:0/2808416656 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): get_auth_request con 0x7f2fa80b4c40 auth_method 0 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fa80b3730 auth_method 0 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.282+0000 7f2fa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.284+0000 7f2fa77fe640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b56c0 0x7f2fa81330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.284+0000 7f2fa77fe640 10 monclient(hunting): get_auth_request con 0x7f2fa80b56c0 auth_method 0 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.284+0000 7f2fa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.284+0000 7f2fa77fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.284+0000 7f2fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.285+0000 7f2fa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.285+0000 7f2fa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.285+0000 7f2fa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 10 monclient(hunting): handle_auth_done global_id 14805 payload 824 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 -- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 msgr2=0x7f2fa8132b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 0x7f2fa8132b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 -- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b56c0 msgr2=0x7f2fa81330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fa80b56c0 0x7f2fa81330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 10 monclient: _finish_hunting 0 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 monclient: found mon.noname-a 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa6ffd640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa8133600 con 0x7f2fa80b3730 2024-07-27T15:17:58.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.286+0000 7f2fa67fc640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2fa80b4c40 0x7f2fa8132b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T15:17:58.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.289+0000 7f2fa6ffd640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fa80b3730 0x7f2fa8144100 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800b440 tx=0x7f2f9800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.289+0000 7f2f8ffff640 10 client.?.objecter ms_handle_connect 0x7f2fa80b3730 2024-07-27T15:17:58.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.289+0000 7f2f8ffff640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98015020 con 0x7f2fa80b3730 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: dump: 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: _finish_auth 0 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: _check_auth_tickets 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f98011830 con 0x7f2fa80b3730 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:58.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98011c20 con 0x7f2fa80b3730 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2f8ffff640 10 monclient: dump: 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 5 monclient: authenticate success, global_id 14805 2024-07-27T15:17:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 10 monclient: _renew_subs 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa813e620 con 0x7f2fa80b3730 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fa57fa640 4 set_mon_vals no callback set 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 10 client.14805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 10 monclient: _renew_subs 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa8138d20 con 0x7f2fa80b3730 2024-07-27T15:17:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.294+0000 7f2fad851640 1 librados: init done 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 10 client.14805.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 15 client.14805.objecter _session_command_op_assign -1 1 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 20 client.14805.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 10 client.14805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.87:3300/0 2024-07-27T15:17:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.295+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2fa8139010 con 0x7f2fa80b3730 2024-07-27T15:17:58.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.315+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2f98018020 con 0x7f2fa80b3730 2024-07-27T15:17:58.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.315+0000 7f2f8ffff640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2f9406d7c0 0x7f2f9406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.315+0000 7f2fa67fc640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2f9406d7c0 0x7f2f9406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.315+0000 7f2fa67fc640 10 monclient: get_auth_request con 0x7f2f9406d7c0 auth_method 0 2024-07-27T15:17:58.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.316+0000 7f2f8ffff640 10 client.14805.objecter ms_handle_connect 0x7f2f9406d7c0 2024-07-27T15:17:58.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.316+0000 7f2fa67fc640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2f9406d7c0 0x7f2f9406fc80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2fa813fa00 tx=0x7f2f9c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f2f980575c0 con 0x7f2fa80b3730 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 10 client.14805.objecter ms_dispatch 0x7f2fa8000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 3 client.14805.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 3 client.14805.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 10 client.14805.objecter checking command 1 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f2f94075530 0x7f2f940779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter _get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 15 client.14805.objecter _session_command_op_remove -1 1 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter _get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 15 client.14805.objecter _session_command_op_assign 5 1 2024-07-27T15:17:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 4 2024-07-27T15:17:58.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 10 client.14805.objecter _send_command 1 2024-07-27T15:17:58.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f98006780 con 0x7f2f94075530 2024-07-27T15:17:58.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 20 client.14805.objecter dump_active .. 0 homeless 2024-07-27T15:17:58.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7f2f980557c0 con 0x7f2fa80b3730 2024-07-27T15:17:58.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.317+0000 7f2f8ffff640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:58.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.318+0000 7f2fa77fe640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f2f94075530 0x7f2f940779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.318+0000 7f2fa77fe640 10 monclient: get_auth_request con 0x7f2f94075530 auth_method 0 2024-07-27T15:17:58.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.321+0000 7f2fa77fe640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f2f94075530 0x7f2f940779d0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.321+0000 7f2f8ffff640 10 client.14805.objecter ms_handle_connect 0x7f2f94075530 2024-07-27T15:17:58.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.345+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== osd.5 v2:172.21.15.112:6810/3751677016 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2f98006780 con 0x7f2f94075530 2024-07-27T15:17:58.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.345+0000 7f2f8ffff640 10 client.14805.objecter ms_dispatch 0x7f2fa8000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:58.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.345+0000 7f2f8ffff640 10 client.14805.objecter _finish_command 1 = system:0 2024-07-27T15:17:58.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.345+0000 7f2f8ffff640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 4 2024-07-27T15:17:58.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.345+0000 7f2f8ffff640 15 client.14805.objecter _session_command_op_remove 5 1 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 10 client.14805.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 15 client.14805.objecter _session_command_op_assign -1 2 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 20 client.14805.objecter _get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 20 client.14805.objecter _get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 15 client.14805.objecter _session_command_op_remove -1 2 2024-07-27T15:17:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 20 client.14805.objecter get_session s=0x7f2f94074e60 osd=5 3 2024-07-27T15:17:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 15 client.14805.objecter _session_command_op_assign 5 2 2024-07-27T15:17:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 4 2024-07-27T15:17:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 10 client.14805.objecter _send_command 2 2024-07-27T15:17:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.358+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 --> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2fa80b3b80 con 0x7f2f94075530 2024-07-27T15:17:58.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2f8ffff640 1 -- 172.21.15.87:0/2808416656 <== osd.5 v2:172.21.15.112:6810/3751677016 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2fa80b3b80 con 0x7f2f94075530 2024-07-27T15:17:58.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2f8ffff640 10 client.14805.objecter ms_dispatch 0x7f2fa8000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2f8ffff640 10 client.14805.objecter _finish_command 2 = system:0 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2f8ffff640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 4 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2f8ffff640 15 client.14805.objecter _session_command_op_remove 5 2 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:58.376 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:58.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 10 librados: watch_flush enter 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 10 librados: watch_flush exit 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 10 client.14805.objecter close_session for osd.5 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f2f94075530 msgr2=0x7f2f940779d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6810/3751677016,v1:172.21.15.112:6811/3751677016] conn(0x7f2f94075530 0x7f2f940779d0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 20 client.14805.objecter put_session s=0x7f2f94074e60 osd=5 1 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 20 client.14805.objecter shutdown clearing up homeless session... 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 10 client.14805.objecter successfully canceled tick 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2f9406d7c0 msgr2=0x7f2f9406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f2f9406d7c0 0x7f2f9406fc80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2fa813fa00 tx=0x7f2f9c008030 comp rx=0 tx=0).stop 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 10 monclient: shutdown 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:58.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fa80b3730 msgr2=0x7f2fa8144100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 --2- 172.21.15.87:0/2808416656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fa80b3730 0x7f2fa8144100 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800b440 tx=0x7f2f9800b900 comp rx=0 tx=0).stop 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fa77fe640 1 -- 172.21.15.87:0/2808416656 reap_dead start 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 shutdown_connections 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.374+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 >> 172.21.15.87:0/2808416656 conn(0x7f2fa8073130 msgr2=0x7f2fa813c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.375+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 shutdown_connections 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.375+0000 7f2fad851640 1 -- 172.21.15.87:0/2808416656 wait complete. 2024-07-27T15:17:58.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.375+0000 7f2fad851640 1 librados: shutdown 2024-07-27T15:17:58.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-07-27T15:17:58.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.487+0000 7f8aad192640 10 monclient: get_monmap_and_config 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.487+0000 7f8aad192640 10 monclient: build_initial_monmap 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.487+0000 7f8aad192640 10 monclient: monmap: 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:58.488480+0000 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:58.488480+0000 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:58.489 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:58.490 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 Processor -- start 2024-07-27T15:17:58.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 -- start start 2024-07-27T15:17:58.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient: init 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 0x7f8aa80ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa80b0380 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8aa80ac6e0 0x7f8aa813c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa80ac6e0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa813c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa80aeb30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa80b0760 con 0x7f8aa80b0380 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa80ae610 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa813cb20 con 0x7f8aa80aeb30 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.488+0000 7f8aad192640 10 monclient(hunting): authenticate will time out at 3418.514713s 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 0x7f8aa80ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa659c640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8aa80ac6e0 0x7f8aa813c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa813c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa813c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37230/0 (socket says 172.21.15.87:37230) 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 1 -- 172.21.15.87:0/1882551933 learned_addr learned my addr 172.21.15.87:0/1882551933 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 0x7f8aa80ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33604/0 (socket says 172.21.15.87:33604) 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 10 monclient(hunting): get_auth_request con 0x7f8aa80aeb30 auth_method 0 2024-07-27T15:17:58.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa759e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 10 monclient(hunting): get_auth_request con 0x7f8aa80b0380 auth_method 0 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa659c640 10 monclient(hunting): get_auth_request con 0x7f8aa80ac6e0 auth_method 0 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.489+0000 7f8aa659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.490+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 10 monclient(hunting): handle_auth_done global_id 21958 payload 274 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 -- 172.21.15.87:0/1882551933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 msgr2=0x7f8aa813c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 --2- 172.21.15.87:0/1882551933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa813c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 -- 172.21.15.87:0/1882551933 >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 msgr2=0x7f8aa80ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 --2- 172.21.15.87:0/1882551933 >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 0x7f8aa80ac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 monclient: found mon.noname-c 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.491+0000 7f8aa659c640 1 -- 172.21.15.87:0/1882551933 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa8073e90 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.493+0000 7f8aa659c640 1 --2- 172.21.15.87:0/1882551933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa813c0a0 secure :-1 s=READY pgs=5043 cs=0 l=1 rev1=1 crypto rx=0x7f8aa806e4b0 tx=0x7f8a9002e0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3d17aa18c50cc69 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 1 -- 172.21.15.87:0/1882551933 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9003b070 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:58.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: dump: 2024-07-27T15:17:58.500 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: _finish_auth 0 2024-07-27T15:17:58.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: _check_auth_tickets 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 1 -- 172.21.15.87:0/1882551933 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9002ee50 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 1 -- 172.21.15.87:0/1882551933 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90036660 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aad192640 5 monclient: authenticate success, global_id 21958 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aad192640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aad192640 4 set_mon_vals no callback set 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.499+0000 7f8aa5d9b640 10 monclient: dump: 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.502 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.503 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 10 monclient: shutdown 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 -- 172.21.15.87:0/1882551933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 msgr2=0x7f8aa813c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 --2- 172.21.15.87:0/1882551933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa813c0a0 secure :-1 s=READY pgs=5043 cs=0 l=1 rev1=1 crypto rx=0x7f8aa806e4b0 tx=0x7f8a9002e0a0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 -- 172.21.15.87:0/1882551933 shutdown_connections 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 --2- 172.21.15.87:0/1882551933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa813c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 --2- 172.21.15.87:0/1882551933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa813c0a0 unknown :-1 s=CLOSED pgs=5043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 --2- 172.21.15.87:0/1882551933 >> v2:172.21.15.87:3300/0 conn(0x7f8aa80b0380 0x7f8aa80ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 -- 172.21.15.87:0/1882551933 >> 172.21.15.87:0/1882551933 conn(0x7f8aa80b4d30 msgr2=0x7f8aa80b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:58.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 -- 172.21.15.87:0/1882551933 shutdown_connections 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 1 -- 172.21.15.87:0/1882551933 wait complete. 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 10 monclient: build_initial_monmap 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.500+0000 7f8aad192640 10 monclient: monmap: 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 librados: starting msgr at 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 librados: starting objecter 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 Processor -- start 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 -- start start 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 librados: setting wanted keys 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 1 librados: calling monclient init 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.501+0000 7f8aad192640 10 monclient: init 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T15:17:58.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa80ac6e0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa80aeb30 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa80b0380 0x7f8aa81d4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa80b0380 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa80ae230 con 0x7f8aa80b0380 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa80abb50 con 0x7f8aa80ac6e0 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa80ae3b0 con 0x7f8aa80aeb30 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:58.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aad192640 10 monclient(hunting): authenticate will time out at 3418.528118s 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aa659c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aa659c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37242/0 (socket says 172.21.15.87:37242) 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aa659c640 1 -- 172.21.15.87:0/3517235755 learned_addr learned my addr 172.21.15.87:0/3517235755 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.502+0000 7f8aa659c640 10 monclient(hunting): get_auth_request con 0x7f8aa80aeb30 auth_method 0 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa759e640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa80b0380 0x7f8aa81d4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa759e640 10 monclient(hunting): get_auth_request con 0x7f8aa80b0380 auth_method 0 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa759e640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa759e640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa6d9d640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa6d9d640 10 monclient(hunting): get_auth_request con 0x7f8aa80ac6e0 auth_method 0 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.503+0000 7f8aa6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.504+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.504+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.504+0000 7f8aa6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.505+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.505+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:58.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.505+0000 7f8aa759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 10 monclient(hunting): handle_auth_done global_id 15554 payload 824 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 -- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 msgr2=0x7f8aa806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 -- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa80b0380 msgr2=0x7f8aa81d4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa80b0380 0x7f8aa81d4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 monclient: found mon.noname-c 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T15:17:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.506+0000 7f8aa659c640 1 -- 172.21.15.87:0/3517235755 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa8070020 con 0x7f8aa80aeb30 2024-07-27T15:17:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.508+0000 7f8aa659c640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000bb30 tx=0x7f8a900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.508+0000 7f8a9f7fe640 10 client.?.objecter ms_handle_connect 0x7f8aa80aeb30 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.508+0000 7f8a9f7fe640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90043020 con 0x7f8aa80aeb30 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: dump: 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.515 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: _finish_auth 0 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: _check_auth_tickets 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9003f410 con 0x7f8aa80aeb30 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9003b040 con 0x7f8aa80aeb30 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 5 monclient: authenticate success, global_id 15554 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 10 monclient: _renew_subs 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa81402a0 con 0x7f8aa80aeb30 2024-07-27T15:17:58.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aa559a640 4 set_mon_vals no callback set 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8a9f7fe640 10 monclient: dump: 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 10 client.15554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 10 monclient: _renew_subs 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa8142fb0 con 0x7f8aa80aeb30 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.512+0000 7f8aad192640 1 librados: init done 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.513+0000 7f8aad192640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.513+0000 7f8aad192640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.513+0000 7f8aad192640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:17:58.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.513+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa80accc0 con 0x7f8aa80aeb30 2024-07-27T15:17:58.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.767+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a90046020 con 0x7f8aa80aeb30 2024-07-27T15:17:58.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.767+0000 7f8a9f7fe640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 0x7f8a8806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:58.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.767+0000 7f8aa6d9d640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 0x7f8a8806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:58.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.767+0000 7f8aa6d9d640 10 monclient: get_auth_request con 0x7f8a8806d7c0 auth_method 0 2024-07-27T15:17:58.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.768+0000 7f8aa6d9d640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 0x7f8a8806fc80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800aba0 tx=0x7f8a98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:58.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.768+0000 7f8a9f7fe640 10 client.15554.objecter ms_handle_connect 0x7f8a8806d7c0 2024-07-27T15:17:58.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f8a90079810 con 0x7f8aa80aeb30 2024-07-27T15:17:58.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 10 client.15554.objecter ms_dispatch 0x7f8aa8000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 3 client.15554.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 3 client.15554.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 20 client.15554.objecter dump_active .. 0 homeless 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f8aa80accc0 con 0x7f8aa80aeb30 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:58.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.781+0000 7f8a9f7fe640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:58.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.960+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8aa80ac1a0 con 0x7f8a8806d7c0 2024-07-27T15:17:58.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.960+0000 7f8a9f7fe640 1 -- 172.21.15.87:0/3517235755 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8aa80ac1a0 con 0x7f8a8806d7c0 2024-07-27T15:17:58.961 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 5 to deep-scrub 2024-07-27T15:17:58.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 10 librados: watch_flush enter 2024-07-27T15:17:58.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 10 librados: watch_flush exit 2024-07-27T15:17:58.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 20 client.15554.objecter shutdown clearing up homeless session... 2024-07-27T15:17:58.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 10 client.15554.objecter successfully canceled tick 2024-07-27T15:17:58.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 msgr2=0x7f8a8806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 0x7f8a8806fc80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800aba0 tx=0x7f8a98008040 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 10 monclient: shutdown 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 msgr2=0x7f8aa81d4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000bb30 tx=0x7f8a900046c0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 shutdown_connections 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f8a8806d7c0 0x7f8a8806fc80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa80b0380 0x7f8aa81d4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa80aeb30 0x7f8aa81d4520 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 --2- 172.21.15.87:0/3517235755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8aa80ac6e0 0x7f8aa806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.962+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 >> 172.21.15.87:0/3517235755 conn(0x7f8aa80b4d30 msgr2=0x7f8aa806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.963+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 shutdown_connections 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.963+0000 7f8aad192640 1 -- 172.21.15.87:0/3517235755 wait complete. 2024-07-27T15:17:58.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:58.963+0000 7f8aad192640 1 librados: shutdown 2024-07-27T15:17:58.976 INFO:tasks.ceph:Scrubbing osd.6 2024-07-27T15:17:58.976 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:59.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.082+0000 7f1616cbe640 10 monclient: get_monmap_and_config 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.082+0000 7f1616cbe640 10 monclient: build_initial_monmap 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.082+0000 7f1616cbe640 10 monclient: monmap: 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:59.083845+0000 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:59.083845+0000 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:59.084 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:59.085 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:59.085 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.083+0000 7f1616cbe640 1 Processor -- start 2024-07-27T15:17:59.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.083+0000 7f1616cbe640 1 -- start start 2024-07-27T15:17:59.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.083+0000 7f1616cbe640 10 monclient: init 2024-07-27T15:17:59.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:59.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f16100b72c0 0x7f16100b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-a con 0x7f16100b72c0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1610072b00 addr v1:172.21.15.87:6789/0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f161013fca0 addr v1:172.21.15.112:6789/0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100ae9a0 con 0x7f1610072b00 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100b1710 con 0x7f161013fca0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f16100b76a0 con 0x7f16100b72c0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1616cbe640 10 monclient(hunting): authenticate will time out at 3419.110400s 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f1614a5a640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f16100b72c0 0x7f16100b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f161525b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f161013fca0 0x7f1610140110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52954/0 (socket says 172.21.15.87:52954) 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.084+0000 7f161525b640 1 -- 172.21.15.87:0/1893952208 learned_addr learned my addr 172.21.15.87:0/1893952208 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.085+0000 7f1614a5a640 10 monclient(hunting): get_auth_request con 0x7f16100b72c0 auth_method 0 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.085+0000 7f1614a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.085+0000 7f1614a5a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.085+0000 7f1614a5a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.086+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.086+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.086+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 1 -- 172.21.15.87:0/1893952208 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693617381 0 0) 0x7f16100b1710 con 0x7f161013fca0 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 10 monclient(hunting): _negotiate 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 10 monclient(hunting): my global_id is 21964 2024-07-27T15:17:59.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.087+0000 7f160f7fe640 1 -- 172.21.15.87:0/1893952208 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1600003520 con 0x7f161013fca0 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 10 monclient(hunting): handle_auth_done global_id 14814 payload 274 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 1 -- 172.21.15.87:0/1893952208 >> v1:172.21.15.112:6789/0 conn(0x7f161013fca0 legacy=0x7f1610140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 1 -- 172.21.15.87:0/1893952208 >> v1:172.21.15.87:6789/0 conn(0x7f1610072b00 legacy=0x7f1610142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 10 monclient: _finish_hunting 0 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 1 monclient: found mon.noname-a 2024-07-27T15:17:59.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:59.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.87:3300/0 2024-07-27T15:17:59.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.089+0000 7f1614a5a640 1 -- 172.21.15.87:0/1893952208 --> v2:172.21.15.87:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16100b12c0 con 0x7f16100b72c0 2024-07-27T15:17:59.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.092+0000 7f1614a5a640 1 --2- 172.21.15.87:0/1893952208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16100b72c0 0x7f16100b1c30 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f16040097a0 tx=0x7f160402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6006dc3b44e8312 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 1 -- 172.21.15.87:0/1893952208 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604035020 con 0x7f16100b72c0 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 10 monclient: dump: 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.097 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 10 monclient: _finish_auth 0 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 10 monclient: _check_auth_tickets 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.095+0000 7f160f7fe640 1 -- 172.21.15.87:0/1893952208 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1604031830 con 0x7f16100b72c0 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f160f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f160f7fe640 1 -- 172.21.15.87:0/1893952208 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604031c00 con 0x7f16100b72c0 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 5 monclient: authenticate success, global_id 14814 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f160f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 4 set_mon_vals no callback set 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f160f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f160f7fe640 10 monclient: dump: 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.099 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 10 monclient: shutdown 2024-07-27T15:17:59.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:59.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 -- 172.21.15.87:0/1893952208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16100b72c0 msgr2=0x7f16100b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 --2- 172.21.15.87:0/1893952208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16100b72c0 0x7f16100b1c30 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f16040097a0 tx=0x7f160402b8d0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 -- 172.21.15.87:0/1893952208 shutdown_connections 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 --2- 172.21.15.87:0/1893952208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16100b72c0 0x7f16100b1c30 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 -- 172.21.15.87:0/1893952208 >> 172.21.15.87:0/1893952208 conn(0x7f1610067d70 msgr2=0x7f16100b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 -- 172.21.15.87:0/1893952208 shutdown_connections 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.096+0000 7f1616cbe640 1 -- 172.21.15.87:0/1893952208 wait complete. 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 10 monclient: build_initial_monmap 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 10 monclient: monmap: 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:59.107 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 librados: starting msgr at 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 librados: starting objecter 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 Processor -- start 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 -- start start 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 librados: setting wanted keys 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 1 librados: calling monclient init 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.097+0000 7f1616cbe640 10 monclient: init 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:59.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1610072b00 0x7f1610147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-a con 0x7f1610072b00 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 0x7f1610148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-b con 0x7f16100b72c0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f161013fca0 0x7f16101488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): picked mon.noname-c con 0x7f161013fca0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f161014e5d0 con 0x7f1610072b00 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16100b2170 con 0x7f16100b72c0 2024-07-27T15:17:59.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f161014e750 con 0x7f161013fca0 2024-07-27T15:17:59.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.098+0000 7f1616cbe640 10 monclient(hunting): authenticate will time out at 3419.124593s 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 0x7f1610148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f1614a5a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1610072b00 0x7f1610147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f161525b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f161013fca0 0x7f16101488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 0x7f1610148380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45524/0 (socket says 172.21.15.87:45524) 2024-07-27T15:17:59.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 1 -- 172.21.15.87:0/1700337548 learned_addr learned my addr 172.21.15.87:0/1700337548 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): get_auth_request con 0x7f16100b72c0 auth_method 0 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f1614a5a640 10 monclient(hunting): get_auth_request con 0x7f1610072b00 auth_method 0 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f1614a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f1614a5a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f1614a5a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f161525b640 10 monclient(hunting): get_auth_request con 0x7f161013fca0 auth_method 0 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f161525b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f161525b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f161525b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.099+0000 7f160ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f1614a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f161525b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f161525b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.100+0000 7f161525b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 10 monclient(hunting): handle_auth_done global_id 21967 payload 824 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 -- 172.21.15.87:0/1700337548 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f161013fca0 msgr2=0x7f16101488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f161013fca0 0x7f16101488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 -- 172.21.15.87:0/1700337548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1610072b00 msgr2=0x7f1610147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1610072b00 0x7f1610147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 10 monclient: _finish_hunting 0 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 monclient: found mon.noname-b 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.102+0000 7f160ffff640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1610149810 con 0x7f16100b72c0 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.104+0000 7f160ffff640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 0x7f1610148380 secure :-1 s=READY pgs=5047 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c400 tx=0x7f15fc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.104+0000 7f160d7fa640 10 client.?.objecter ms_handle_connect 0x7f16100b72c0 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.104+0000 7f160d7fa640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc0052d0 con 0x7f16100b72c0 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: dump: 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.118 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: _finish_auth 0 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: _check_auth_tickets 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15fc005490 con 0x7f16100b72c0 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc007260 con 0x7f16100b72c0 2024-07-27T15:17:59.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160d7fa640 10 monclient: dump: 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.120 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.121 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 5 monclient: authenticate success, global_id 21967 2024-07-27T15:17:59.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 10 monclient: _renew_subs 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1610149ae0 con 0x7f16100b72c0 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f160effd640 4 set_mon_vals no callback set 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 10 client.21967.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 10 monclient: _renew_subs 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161014a460 con 0x7f16100b72c0 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.108+0000 7f1616cbe640 1 librados: init done 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 10 client.21967.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 15 client.21967.objecter _session_command_op_assign -1 1 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 20 client.21967.objecter _get_session osd=-1 returning homeless 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 10 client.21967.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:59.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.109+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1610148eb0 con 0x7f16100b72c0 2024-07-27T15:17:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.128+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f15fc007400 con 0x7f16100b72c0 2024-07-27T15:17:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.128+0000 7f160d7fa640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f15f406d810 0x7f15f406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.128+0000 7f1614a5a640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f15f406d810 0x7f15f406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.128+0000 7f1614a5a640 10 monclient: get_auth_request con 0x7f15f406d810 auth_method 0 2024-07-27T15:17:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.129+0000 7f1614a5a640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f15f406d810 0x7f15f406fcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1604036000 tx=0x7f1604008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.129+0000 7f160d7fa640 10 client.21967.objecter ms_handle_connect 0x7f15f406d810 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f15fc056ed0 con 0x7f16100b72c0 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 10 client.21967.objecter ms_dispatch 0x7f1610000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 3 client.21967.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 3 client.21967.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 10 client.21967.objecter checking command 1 2024-07-27T15:17:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f15f40754f0 0x7f15f4077990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter _get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 15 client.21967.objecter _session_command_op_remove -1 1 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter _get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 15 client.21967.objecter _session_command_op_assign 6 1 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 4 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 10 client.21967.objecter _send_command 1 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15fc010e50 con 0x7f15f40754f0 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 20 client.21967.objecter dump_active .. 0 homeless 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7f15fc0550d0 con 0x7f16100b72c0 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.131+0000 7f160d7fa640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:17:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.132+0000 7f161525b640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f15f40754f0 0x7f15f4077990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.132+0000 7f161525b640 10 monclient: get_auth_request con 0x7f15f40754f0 auth_method 0 2024-07-27T15:17:59.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.135+0000 7f161525b640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f15f40754f0 0x7f15f4077990 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.135+0000 7f160d7fa640 10 client.21967.objecter ms_handle_connect 0x7f15f40754f0 2024-07-27T15:17:59.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.158+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== osd.6 v2:172.21.15.112:6802/591715754 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15fc010e50 con 0x7f15f40754f0 2024-07-27T15:17:59.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.158+0000 7f160d7fa640 10 client.21967.objecter ms_dispatch 0x7f1610000b90 command_reply(tid 1: 0 ) 2024-07-27T15:17:59.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.158+0000 7f160d7fa640 10 client.21967.objecter _finish_command 1 = system:0 2024-07-27T15:17:59.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.158+0000 7f160d7fa640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 4 2024-07-27T15:17:59.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.158+0000 7f160d7fa640 15 client.21967.objecter _session_command_op_remove 6 1 2024-07-27T15:17:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 10 client.21967.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:17:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 15 client.21967.objecter _session_command_op_assign -1 2 2024-07-27T15:17:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 20 client.21967.objecter _get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 20 client.21967.objecter _get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 15 client.21967.objecter _session_command_op_remove -1 2 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 20 client.21967.objecter get_session s=0x7f15f4074e20 osd=6 3 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 15 client.21967.objecter _session_command_op_assign 6 2 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 4 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 10 client.21967.objecter _send_command 2 2024-07-27T15:17:59.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.171+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 --> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f16100b1c30 con 0x7f15f40754f0 2024-07-27T15:17:59.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f160d7fa640 1 -- 172.21.15.87:0/1700337548 <== osd.6 v2:172.21.15.112:6802/591715754 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f16100b1c30 con 0x7f15f40754f0 2024-07-27T15:17:59.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f160d7fa640 10 client.21967.objecter ms_dispatch 0x7f1610000b90 command_reply(tid 2: 0 ) 2024-07-27T15:17:59.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f160d7fa640 10 client.21967.objecter _finish_command 2 = system:0 2024-07-27T15:17:59.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f160d7fa640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 4 2024-07-27T15:17:59.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f160d7fa640 15 client.21967.objecter _session_command_op_remove 6 2 2024-07-27T15:17:59.188 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:17:59.188 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:17:59.188 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:17:59.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 10 librados: watch_flush enter 2024-07-27T15:17:59.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 10 librados: watch_flush exit 2024-07-27T15:17:59.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 10 client.21967.objecter close_session for osd.6 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f15f40754f0 msgr2=0x7f15f4077990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6802/591715754,v1:172.21.15.112:6803/591715754] conn(0x7f15f40754f0 0x7f15f4077990 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 20 client.21967.objecter put_session s=0x7f15f4074e20 osd=6 1 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 20 client.21967.objecter shutdown clearing up homeless session... 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 10 client.21967.objecter successfully canceled tick 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f15f406d810 msgr2=0x7f15f406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f15f406d810 0x7f15f406fcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1604036000 tx=0x7f1604008030 comp rx=0 tx=0).stop 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 10 monclient: shutdown 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 msgr2=0x7f1610148380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 --2- 172.21.15.87:0/1700337548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16100b72c0 0x7f1610148380 secure :-1 s=READY pgs=5047 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c400 tx=0x7f15fc00c8c0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f161525b640 1 -- 172.21.15.87:0/1700337548 reap_dead start 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 shutdown_connections 2024-07-27T15:17:59.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 >> 172.21.15.87:0/1700337548 conn(0x7f1610067d70 msgr2=0x7f16100b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:59.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 shutdown_connections 2024-07-27T15:17:59.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.186+0000 7f1616cbe640 1 -- 172.21.15.87:0/1700337548 wait complete. 2024-07-27T15:17:59.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.187+0000 7f1616cbe640 1 librados: shutdown 2024-07-27T15:17:59.198 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-07-27T15:17:59.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.304+0000 7f6188f9d640 10 monclient: get_monmap_and_config 2024-07-27T15:17:59.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.304+0000 7f6188f9d640 10 monclient: build_initial_monmap 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.304+0000 7f6188f9d640 10 monclient: monmap: 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:59.305423+0000 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:59.305423+0000 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:59.306 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:59.307 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:59.307 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.304+0000 7f6188f9d640 1 Processor -- start 2024-07-27T15:17:59.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 -- start start 2024-07-27T15:17:59.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient: init 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7f6184072b70 0x7f61840b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f6184072b70 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840af970 0x7f61840b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f61840af970 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f61840b1c80 0x7f61840b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f61840b1c80 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:59.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7f6184072f50 con 0x7f6184072b70 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f61840aff60 con 0x7f61840b1c80 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f61840b3060 con 0x7f61840af970 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.305+0000 7f6188f9d640 10 monclient(hunting): authenticate will time out at 3419.331812s 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6181d9b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840af970 0x7f61840b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f61840b1c80 0x7f61840b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f61840b1c80 0x7f61840b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45536/0 (socket says 172.21.15.87:45536) 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 1 -- 172.21.15.87:0/958384650 learned_addr learned my addr 172.21.15.87:0/958384650 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:59.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 10 monclient(hunting): get_auth_request con 0x7f61840b1c80 auth_method 0 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6182d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6181d9b640 10 monclient(hunting): get_auth_request con 0x7f61840af970 auth_method 0 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6181d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.306+0000 7f6181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6181d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6181d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.307+0000 7f6181d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 10 monclient(hunting): handle_auth_done global_id 21970 payload 274 2024-07-27T15:17:59.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 -- 172.21.15.87:0/958384650 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840af970 msgr2=0x7f61840b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 --2- 172.21.15.87:0/958384650 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840af970 0x7f61840b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 -- 172.21.15.87:0/958384650 >> v2:172.21.15.87:3300/0 conn(0x7f6184072b70 msgr2=0x7f61840b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 --2- 172.21.15.87:0/958384650 >> v2:172.21.15.87:3300/0 conn(0x7f6184072b70 0x7f61840b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 10 monclient: _finish_hunting 0 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 monclient: found mon.noname-c 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.308+0000 7f6182d9d640 1 -- 172.21.15.87:0/958384650 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618413fda0 con 0x7f61840b1c80 2024-07-27T15:17:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.315+0000 7f6182d9d640 1 --2- 172.21.15.87:0/958384650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61840b1c80 0x7f61840b2110 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 crypto rx=0x7f6184073800 tx=0x7f617c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=210297fa5e85816c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 1 -- 172.21.15.87:0/958384650 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c02ca70 con 0x7f61840b1c80 2024-07-27T15:17:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:17:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: dump: 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: _finish_auth 0 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: _check_auth_tickets 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 1 -- 172.21.15.87:0/958384650 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617c02cc10 con 0x7f61840b1c80 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f6188f9d640 5 monclient: authenticate success, global_id 21970 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f6188f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f6188f9d640 10 monclient: get_monmap_and_config success 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f6188f9d640 4 set_mon_vals no callback set 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 1 -- 172.21.15.87:0/958384650 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c0343f0 con 0x7f61840b1c80 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.319+0000 7f618159a640 10 monclient: dump: 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.323 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 10 monclient: shutdown 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 -- 172.21.15.87:0/958384650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61840b1c80 msgr2=0x7f61840b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 --2- 172.21.15.87:0/958384650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61840b1c80 0x7f61840b2110 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 crypto rx=0x7f6184073800 tx=0x7f617c02bed0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 -- 172.21.15.87:0/958384650 shutdown_connections 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 --2- 172.21.15.87:0/958384650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61840b1c80 0x7f61840b2110 unknown :-1 s=CLOSED pgs=5048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 --2- 172.21.15.87:0/958384650 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840af970 0x7f61840b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 --2- 172.21.15.87:0/958384650 >> v2:172.21.15.87:3300/0 conn(0x7f6184072b70 0x7f61840b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 -- 172.21.15.87:0/958384650 >> 172.21.15.87:0/958384650 conn(0x7f6184068150 msgr2=0x7f61840b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 -- 172.21.15.87:0/958384650 shutdown_connections 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.320+0000 7f6188f9d640 1 -- 172.21.15.87:0/958384650 wait complete. 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 10 monclient: build_initial_monmap 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 10 monclient: monmap: 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:17:59.327 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 librados: starting msgr at 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 librados: starting objecter 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 Processor -- start 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 -- start start 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 librados: setting wanted keys 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 1 librados: calling monclient init 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.321+0000 7f6188f9d640 10 monclient: init 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f6184072b70 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61840af970 0x7f61840ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f61840af970 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 0x7f61840ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f61840b1c80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6184140540 con 0x7f61840af970 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61841403c0 con 0x7f6184072b70 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f61841406c0 con 0x7f61840b1c80 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6188f9d640 10 monclient(hunting): authenticate will time out at 3419.348395s 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.322+0000 7f6182d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 0x7f61840ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45546/0 (socket says 172.21.15.87:45546) 2024-07-27T15:17:59.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 0x7f61840ae8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37280/0 (socket says 172.21.15.87:37280) 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 1 -- 172.21.15.87:0/1586589938 learned_addr learned my addr 172.21.15.87:0/1586589938 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): get_auth_request con 0x7f6184072b70 auth_method 0 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): get_auth_request con 0x7f61840b1c80 auth_method 0 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f618259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.323+0000 7f6182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 10 monclient(hunting): handle_auth_done global_id 21973 payload 824 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 -- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 msgr2=0x7f61840ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 0x7f61840ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 -- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61840af970 msgr2=0x7f61840ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61840af970 0x7f61840ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 10 monclient: _finish_hunting 0 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 monclient: found mon.noname-b 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.326+0000 7f618259c640 1 -- 172.21.15.87:0/1586589938 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61840aee10 con 0x7f6184072b70 2024-07-27T15:17:59.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.329+0000 7f618259c640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 secure :-1 s=READY pgs=5049 cs=0 l=1 rev1=1 crypto rx=0x7f617400b4d0 tx=0x7f617400b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.329+0000 7f6172ffd640 10 client.?.objecter ms_handle_connect 0x7f6184072b70 2024-07-27T15:17:59.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.329+0000 7f6172ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174014070 con 0x7f6184072b70 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: dump: 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.334 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: _finish_auth 0 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 5 monclient: authenticate success, global_id 21973 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174010870 con 0x7f6184072b70 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174010c60 con 0x7f6184072b70 2024-07-27T15:17:59.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 10 monclient: _renew_subs 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61841d8060 con 0x7f6184072b70 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6172ffd640 10 monclient: dump: 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:17:59.336 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6180d99640 4 set_mon_vals no callback set 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 10 client.21973.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 10 monclient: _renew_subs 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61841d8830 con 0x7f6184072b70 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.332+0000 7f6188f9d640 1 librados: init done 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.333+0000 7f6188f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.333+0000 7f6188f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:59.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.333+0000 7f6188f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.333+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61840b0410 con 0x7f6184072b70 2024-07-27T15:17:59.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.351+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6174017070 con 0x7f6184072b70 2024-07-27T15:17:59.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.351+0000 7f6172ffd640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 0x7f616406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.351+0000 7f6181d9b640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 0x7f616406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.351+0000 7f6181d9b640 10 monclient: get_auth_request con 0x7f616406d810 auth_method 0 2024-07-27T15:17:59.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.352+0000 7f6181d9b640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 0x7f616406fcd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6184146d50 tx=0x7f61780095a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:17:59.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.352+0000 7f6172ffd640 10 client.21973.objecter ms_handle_connect 0x7f616406d810 2024-07-27T15:17:59.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.354+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f6174057440 con 0x7f6184072b70 2024-07-27T15:17:59.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.354+0000 7f6172ffd640 10 client.21973.objecter ms_dispatch 0x7f6184000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:17:59.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.354+0000 7f6172ffd640 3 client.21973.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:17:59.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.354+0000 7f6172ffd640 3 client.21973.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:17:59.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.355+0000 7f6172ffd640 20 client.21973.objecter dump_active .. 0 homeless 2024-07-27T15:17:59.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.609+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f61740593a0 con 0x7f6184072b70 2024-07-27T15:17:59.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.609+0000 7f6172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:17:59.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.609+0000 7f6172ffd640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:17:59.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.790+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f61841438e0 con 0x7f616406d810 2024-07-27T15:17:59.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.790+0000 7f6172ffd640 1 -- 172.21.15.87:0/1586589938 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f61841438e0 con 0x7f616406d810 2024-07-27T15:17:59.791 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 6 to deep-scrub 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 10 librados: watch_flush enter 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 10 librados: watch_flush exit 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 20 client.21973.objecter shutdown clearing up homeless session... 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 10 client.21973.objecter successfully canceled tick 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 msgr2=0x7f616406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 0x7f616406fcd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6184146d50 tx=0x7f61780095a0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 10 monclient: shutdown 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 msgr2=0x7f6184146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:17:59.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 secure :-1 s=READY pgs=5049 cs=0 l=1 rev1=1 crypto rx=0x7f617400b4d0 tx=0x7f617400b990 comp rx=0 tx=0).stop 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 shutdown_connections 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f616406d810 0x7f616406fcd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61840b1c80 0x7f61840ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61840af970 0x7f61840ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 --2- 172.21.15.87:0/1586589938 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6184072b70 0x7f6184146500 unknown :-1 s=CLOSED pgs=5049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.792+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 >> 172.21.15.87:0/1586589938 conn(0x7f6184068150 msgr2=0x7f618413cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.793+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 shutdown_connections 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.793+0000 7f6188f9d640 1 -- 172.21.15.87:0/1586589938 wait complete. 2024-07-27T15:17:59.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.793+0000 7f6188f9d640 1 librados: shutdown 2024-07-27T15:17:59.804 INFO:tasks.ceph:Scrubbing osd.7 2024-07-27T15:17:59.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.905+0000 7fc0fccdf640 10 monclient: get_monmap_and_config 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.905+0000 7fc0fccdf640 10 monclient: build_initial_monmap 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.905+0000 7fc0fccdf640 10 monclient: monmap: 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:17:59.906709+0000 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:17:59.906709+0000 2024-07-27T15:17:59.907 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.906+0000 7fc0fccdf640 1 Processor -- start 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.906+0000 7fc0fccdf640 1 -- start start 2024-07-27T15:17:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.906+0000 7fc0fccdf640 10 monclient: init 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient: _reopen_session rank -1 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0f806bc20 0x7fc0f813fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-b con 0x7fc0f806bc20 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-c con 0x7fc0f80b0cd0 addr v2:172.21.15.112:3300/0 2024-07-27T15:17:59.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fc0f81460d0 0x7fc0f8146520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-a con 0x7fc0f81460d0 addr v2:172.21.15.87:3300/0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f81407b0 con 0x7fc0f81460d0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f80aec80 con 0x7fc0f80b0cd0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f806cac0 con 0x7fc0f806bc20 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): _renew_subs 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0fccdf640 10 monclient(hunting): authenticate will time out at 3419.933098s 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0f659c640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0f806bc20 0x7fc0f813fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0f6d9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fc0f81460d0 0x7fc0f8146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.907+0000 7fc0f5d9b640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fc0f81460d0 0x7fc0f8146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33672/0 (socket says 172.21.15.87:33672) 2024-07-27T15:17:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45550/0 (socket says 172.21.15.87:45550) 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/3787170286 learned_addr learned my addr 172.21.15.87:0/3787170286 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request con 0x7fc0f81460d0 auth_method 0 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): get_auth_request con 0x7fc0f80b0cd0 auth_method 0 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): get_auth_request con 0x7fc0f806bc20 auth_method 0 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:17:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:17:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.908+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 10 monclient(hunting): handle_auth_done global_id 21976 payload 274 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 -- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0f806bc20 msgr2=0x7fc0f813fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0f806bc20 0x7fc0f813fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 -- 172.21.15.87:0/3787170286 >> v2:172.21.15.87:3300/0 conn(0x7fc0f81460d0 msgr2=0x7fc0f8146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.87:3300/0 conn(0x7fc0f81460d0 0x7fc0f8146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 10 monclient: _finish_hunting 0 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 monclient: found mon.noname-c 2024-07-27T15:17:59.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.112:3300/0 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.910+0000 7fc0f5d9b640 1 -- 172.21.15.87:0/3787170286 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f806cda0 con 0x7fc0f80b0cd0 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.912+0000 7fc0f5d9b640 1 -- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 msgr2=0x7fc0f8140270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.912+0000 7fc0f5d9b640 1 -- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 msgr2=0x7fc0f8140270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.912+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fc0f80b7740 tx=0x7fc0ec02b840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T15:17:59.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:17:59.912+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T15:18:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f5d9b640 10 monclient: get_auth_request con 0x7fc0f80b0cd0 auth_method 0 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f5d9b640 0 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f559a640 10 monclient: ms_handle_reset current mon v2:172.21.15.112:3300/0 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f559a640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 msgr2=0x7fc0f8140270 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.113+0000 7fc0f559a640 1 --2- 172.21.15.87:0/3787170286 >> v2:172.21.15.112:3300/0 conn(0x7fc0f80b0cd0 0x7fc0f8140270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 10 monclient: _add_conns ranks=[0,1] 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0e0003500 addr v1:172.21.15.87:6789/0 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0e0007490 addr v1:172.21.15.112:6789/0 2024-07-27T15:18:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e000b440 con 0x7fc0e0003500 2024-07-27T15:18:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e000c680 con 0x7fc0e0007490 2024-07-27T15:18:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.114+0000 7fc0f559a640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.112:3300/0 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.116+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698029656 0 0) 0x7fc0e000b440 con 0x7fc0e0003500 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.116+0000 7fc0f559a640 10 monclient(hunting): _negotiate 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.116+0000 7fc0f559a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.116+0000 7fc0f559a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> v1:172.21.15.87:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fc0e000c850 con 0x7fc0e0003500 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.1 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750327726 0 0) 0x7fc0e000c680 con 0x7fc0e0007490 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 10 monclient(hunting): _negotiate 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T15:18:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.117+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> v1:172.21.15.112:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fc0e000b440 con 0x7fc0e0007490 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3449626560 0 0) 0x7fc0e000c850 con 0x7fc0e0003500 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 >> v1:172.21.15.112:6789/0 conn(0x7fc0e0007490 legacy=0x7fc0e0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 monclient: found mon.noname-a-legacy 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.87:6789/0 2024-07-27T15:18:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e000d980 con 0x7fc0e0003500 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0fccdf640 5 monclient: authenticate success, global_id 21976 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0fccdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1560156229 0 0) 0x7fc0e8009880 con 0x7fc0e0003500 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: dump: 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.125 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] but i'm connected to v1:172.21.15.87:6789/0, reconnecting 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 >> v1:172.21.15.87:6789/0 conn(0x7fc0e0003500 legacy=0x7fc0e0005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/3787170286 reap_dead start 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 0x7fc0e00098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): picked mon.c con 0x7fc0e000cf80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000d360 0x7fc0e000b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): picked mon.a con 0x7fc0e000d360 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0e000c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): picked mon.b con 0x7fc0e000bc10 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e000a3e0 con 0x7fc0e000d360 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e000a560 con 0x7fc0e000bc10 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e000a730 con 0x7fc0e000cf80 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0f559a640 10 monclient(hunting): _finish_auth 0 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.120+0000 7fc0fccdf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.121+0000 7fc0f659c640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 0x7fc0e00098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.121+0000 7fc0f659c640 10 monclient(hunting): get_auth_request con 0x7fc0e000cf80 auth_method 0 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.121+0000 7fc0f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.121+0000 7fc0f659c640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.121+0000 7fc0f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0e000c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request con 0x7fc0e000bc10 auth_method 0 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.122+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-27T15:18:00.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.123+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000d360 0x7fc0e000b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 10 monclient(hunting): handle_auth_done global_id 21976 payload 293 2024-07-27T15:18:00.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 -- 172.21.15.87:0/3787170286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 msgr2=0x7fc0e000c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:00.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0e000c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 -- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000d360 msgr2=0x7fc0e000b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:00.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000d360 0x7fc0e000b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 monclient: found mon.c 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 1 -- 172.21.15.87:0/3787170286 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e000c850 con 0x7fc0e000cf80 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.124+0000 7fc0f659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.126+0000 7fc0f659c640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 0x7fc0e00098d0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e380 tx=0x7fc0e000ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e218a4f1778abab7 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8017070 con 0x7fc0e000cf80 2024-07-27T15:18:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: dump: 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0fccdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e8009be0 con 0x7fc0e000cf80 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0fccdf640 10 monclient: get_monmap_and_config success 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0fccdf640 4 set_mon_vals no callback set 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 1 -- 172.21.15.87:0/3787170286 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8009da0 con 0x7fc0e000cf80 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.129+0000 7fc0f559a640 10 monclient: dump: 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 10 monclient: shutdown 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 -- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 msgr2=0x7fc0e00098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 0x7fc0e00098d0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e380 tx=0x7fc0e000ca90 comp rx=0 tx=0).stop 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 -- 172.21.15.87:0/3787170286 shutdown_connections 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0e000c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000d360 0x7fc0e000b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/3787170286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000cf80 0x7fc0e00098d0 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 -- 172.21.15.87:0/3787170286 >> 172.21.15.87:0/3787170286 conn(0x7fc0f8067f70 msgr2=0x7fc0f80b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 -- 172.21.15.87:0/3787170286 shutdown_connections 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.130+0000 7fc0fccdf640 1 -- 172.21.15.87:0/3787170286 wait complete. 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.131+0000 7fc0fccdf640 10 monclient: build_initial_monmap 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.131+0000 7fc0fccdf640 10 monclient: monmap: 2024-07-27T15:18:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.131+0000 7fc0fccdf640 1 librados: starting msgr at 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.131+0000 7fc0fccdf640 1 librados: starting objecter 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 Processor -- start 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 -- start start 2024-07-27T15:18:00.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 librados: setting wanted keys 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 librados: calling monclient init 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient: init 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0f8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e000bc10 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000cf80 0x7fc0f813aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e000cf80 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 0x7fc0f813af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e000d360 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8146520 con 0x7fc0e000cf80 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f806c210 con 0x7fc0e000bc10 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f81466a0 con 0x7fc0e000d360 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.132+0000 7fc0fccdf640 10 monclient(hunting): authenticate will time out at 3420.158684s 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 0x7fc0f813af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 0x7fc0f813af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37300/0 (socket says 172.21.15.87:37300) 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/2014568197 learned_addr learned my addr 172.21.15.87:0/2014568197 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request con 0x7fc0e000d360 auth_method 0 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.133+0000 7fc0f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.134+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.134+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.134+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 10 monclient(hunting): handle_auth_done global_id 15569 payload 824 2024-07-27T15:18:00.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 msgr2=0x7fc0f8070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0e000bc10 0x7fc0f8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/2014568197 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000cf80 msgr2=0x7fc0f813aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0e000cf80 0x7fc0f813aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 monclient: found mon.noname-c 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.138+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f813b4c0 con 0x7fc0e000d360 2024-07-27T15:18:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.141+0000 7fc0f6d9d640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 0x7fc0f813af80 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fc0f81464b0 tx=0x7fc0f000f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.141+0000 7fc0e6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0e000d360 2024-07-27T15:18:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.141+0000 7fc0e6ffd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:18:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0019020 con 0x7fc0e000d360 2024-07-27T15:18:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: dump: 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0f000fb80 con 0x7fc0e000d360 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0003990 con 0x7fc0e000d360 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 5 monclient: authenticate success, global_id 15569 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 monclient: _renew_subs 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f8072ae0 con 0x7fc0e000d360 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0f4d99640 4 set_mon_vals no callback set 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0e6ffd640 10 monclient: dump: 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.148 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 client.15569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 monclient: _renew_subs 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f80730a0 con 0x7fc0e000d360 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 1 librados: init done 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 client.15569.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 15 client.15569.objecter _session_command_op_assign -1 1 2024-07-27T15:18:00.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 20 client.15569.objecter _get_session osd=-1 returning homeless 2024-07-27T15:18:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 client.15569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.145+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0f8071bc0 con 0x7fc0e000d360 2024-07-27T15:18:00.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.166+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc0f001c020 con 0x7fc0e000d360 2024-07-27T15:18:00.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.166+0000 7fc0e6ffd640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc0d806d7c0 0x7fc0d806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.166+0000 7fc0f659c640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc0d806d7c0 0x7fc0d806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.166+0000 7fc0f659c640 10 monclient: get_auth_request con 0x7fc0d806d7c0 auth_method 0 2024-07-27T15:18:00.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.167+0000 7fc0f659c640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc0d806d7c0 0x7fc0d806fc80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8000c00 tx=0x7fc0e8013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.167+0000 7fc0e6ffd640 10 client.15569.objecter ms_handle_connect 0x7fc0d806d7c0 2024-07-27T15:18:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7fc0f0057b80 con 0x7fc0e000d360 2024-07-27T15:18:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 10 client.15569.objecter ms_dispatch 0x7fc0f8000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:18:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 3 client.15569.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:18:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 3 client.15569.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 10 client.15569.objecter checking command 1 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7fc0d8075530 0x7fc0d80779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter _get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 15 client.15569.objecter _session_command_op_remove -1 1 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter _get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 15 client.15569.objecter _session_command_op_assign 7 1 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 4 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 10 client.15569.objecter _send_command 1 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0f0029ea0 con 0x7fc0d8075530 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0e6ffd640 20 client.15569.objecter dump_active .. 0 homeless 2024-07-27T15:18:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.170+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7fc0d8075530 0x7fc0d80779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.171+0000 7fc0f5d9b640 10 monclient: get_auth_request con 0x7fc0d8075530 auth_method 0 2024-07-27T15:18:00.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.172+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=1468) ==== 24+0+0 (secure 0 0 0) 0x7fc0f0055d80 con 0x7fc0e000d360 2024-07-27T15:18:00.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.172+0000 7fc0e6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1468 2024-07-27T15:18:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.174+0000 7fc0f5d9b640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7fc0d8075530 0x7fc0d80779d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.174+0000 7fc0e6ffd640 10 client.15569.objecter ms_handle_connect 0x7fc0d8075530 2024-07-27T15:18:00.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.197+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== osd.7 v2:172.21.15.112:6818/2667521237 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0f0029ea0 con 0x7fc0d8075530 2024-07-27T15:18:00.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.197+0000 7fc0e6ffd640 10 client.15569.objecter ms_dispatch 0x7fc0f8000b90 command_reply(tid 1: 0 ) 2024-07-27T15:18:00.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.197+0000 7fc0e6ffd640 10 client.15569.objecter _finish_command 1 = system:0 2024-07-27T15:18:00.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.197+0000 7fc0e6ffd640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 4 2024-07-27T15:18:00.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.197+0000 7fc0e6ffd640 15 client.15569.objecter _session_command_op_remove 7 1 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 10 client.15569.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 15 client.15569.objecter _session_command_op_assign -1 2 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 20 client.15569.objecter _get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 20 client.15569.objecter _get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 15 client.15569.objecter _session_command_op_remove -1 2 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 20 client.15569.objecter get_session s=0x7fc0d8074e60 osd=7 3 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 15 client.15569.objecter _session_command_op_assign 7 2 2024-07-27T15:18:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 4 2024-07-27T15:18:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 10 client.15569.objecter _send_command 2 2024-07-27T15:18:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.211+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 --> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc0f80712b0 con 0x7fc0d8075530 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0e6ffd640 1 -- 172.21.15.87:0/2014568197 <== osd.7 v2:172.21.15.112:6818/2667521237 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc0f80712b0 con 0x7fc0d8075530 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0e6ffd640 10 client.15569.objecter ms_dispatch 0x7fc0f8000b90 command_reply(tid 2: 0 ) 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0e6ffd640 10 client.15569.objecter _finish_command 2 = system:0 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0e6ffd640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 4 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0e6ffd640 15 client.15569.objecter _session_command_op_remove 7 2 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-07-27T15:18:00.225 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-27T15:18:00.226 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 10 librados: watch_flush enter 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 10 librados: watch_flush exit 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 10 client.15569.objecter close_session for osd.7 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7fc0d8075530 msgr2=0x7fc0d80779d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6818/2667521237,v1:172.21.15.112:6819/2667521237] conn(0x7fc0d8075530 0x7fc0d80779d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 20 client.15569.objecter put_session s=0x7fc0d8074e60 osd=7 1 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 20 client.15569.objecter shutdown clearing up homeless session... 2024-07-27T15:18:00.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 10 client.15569.objecter successfully canceled tick 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc0d806d7c0 msgr2=0x7fc0d806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fc0d806d7c0 0x7fc0d806fc80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8000c00 tx=0x7fc0e8013000 comp rx=0 tx=0).stop 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 10 monclient: shutdown 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 msgr2=0x7fc0f813af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 --2- 172.21.15.87:0/2014568197 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc0e000d360 0x7fc0f813af80 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fc0f81464b0 tx=0x7fc0f000f580 comp rx=0 tx=0).stop 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0f6d9d640 1 -- 172.21.15.87:0/2014568197 reap_dead start 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 shutdown_connections 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 >> 172.21.15.87:0/2014568197 conn(0x7fc0f8067f70 msgr2=0x7fc0f8144eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 shutdown_connections 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 -- 172.21.15.87:0/2014568197 wait complete. 2024-07-27T15:18:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.224+0000 7fc0fccdf640 1 librados: shutdown 2024-07-27T15:18:00.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-07-27T15:18:00.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.342+0000 7fa9bed9e640 10 monclient: get_monmap_and_config 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.342+0000 7fa9bed9e640 10 monclient: build_initial_monmap 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.343+0000 7fa9bed9e640 10 monclient: monmap: 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:18:00.344040+0000 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:18:00.344040+0000 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:18:00.344 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:18:00.345 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:18:00.345 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:18:00.345 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.343+0000 7fa9bed9e640 1 Processor -- start 2024-07-27T15:18:00.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.343+0000 7fa9bed9e640 1 -- start start 2024-07-27T15:18:00.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.343+0000 7fa9bed9e640 10 monclient: init 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b8072b70 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa9b80af970 0x7fa9b80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b80af970 addr v2:172.21.15.87:3300/0 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9b80b1c80 addr v1:172.21.15.87:6789/0 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b80aff60 con 0x7fa9b80b1c80 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b8072f50 con 0x7fa9b80af970 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b80b3060 con 0x7fa9b8072b70 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.344+0000 7fa9bed9e640 10 monclient(hunting): authenticate will time out at 3420.370431s 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9b7fff640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7fa9b80af970 0x7fa9b80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37314/0 (socket says 172.21.15.87:37314) 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 1 -- 172.21.15.87:0/3707400791 learned_addr learned my addr 172.21.15.87:0/3707400791 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 10 monclient(hunting): get_auth_request con 0x7fa9b8072b70 auth_method 0 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9bcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9b7fff640 10 monclient(hunting): get_auth_request con 0x7fa9b80af970 auth_method 0 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9b7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.345+0000 7fa9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.346+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_done global_id 15572 payload 274 2024-07-27T15:18:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 1 -- 172.21.15.87:0/3707400791 >> v2:172.21.15.87:3300/0 conn(0x7fa9b80af970 msgr2=0x7fa9b80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 1 --2- 172.21.15.87:0/3707400791 >> v2:172.21.15.87:3300/0 conn(0x7fa9b80af970 0x7fa9b80b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 1 -- 172.21.15.87:0/3707400791 >> v1:172.21.15.87:6789/0 conn(0x7fa9b80b1c80 legacy=0x7fa9b813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 1 monclient: found mon.noname-b 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.347+0000 7fa9bcb3a640 1 -- 172.21.15.87:0/3707400791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b81404b0 con 0x7fa9b8072b70 2024-07-27T15:18:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.349+0000 7fa9bcb3a640 1 --2- 172.21.15.87:0/3707400791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009870 tx=0x7fa9ac02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d688ed8c9526fca9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 1 -- 172.21.15.87:0/3707400791 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac035020 con 0x7fa9b8072b70 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: dump: 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.354 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 1 -- 172.21.15.87:0/3707400791 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac031640 con 0x7fa9b8072b70 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 1 -- 172.21.15.87:0/3707400791 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac02d040 con 0x7fa9b8072b70 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9bed9e640 5 monclient: authenticate success, global_id 15572 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9bed9e640 10 monclient: get_monmap_and_config success 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9bed9e640 4 set_mon_vals no callback set 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.353+0000 7fa9b77fe640 10 monclient: dump: 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.356 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.357 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.357 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.357 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 10 monclient: shutdown 2024-07-27T15:18:00.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:00.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 -- 172.21.15.87:0/3707400791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 msgr2=0x7fa9b80b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/3707400791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009870 tx=0x7fa9ac02b8f0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 -- 172.21.15.87:0/3707400791 shutdown_connections 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/3707400791 >> v2:172.21.15.87:3300/0 conn(0x7fa9b80af970 0x7fa9b80b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/3707400791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b8072b70 0x7fa9b80b4ca0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 -- 172.21.15.87:0/3707400791 >> 172.21.15.87:0/3707400791 conn(0x7fa9b8068150 msgr2=0x7fa9b80b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 -- 172.21.15.87:0/3707400791 shutdown_connections 2024-07-27T15:18:00.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 1 -- 172.21.15.87:0/3707400791 wait complete. 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 10 monclient: build_initial_monmap 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.354+0000 7fa9bed9e640 10 monclient: monmap: 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:18:00.362 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 librados: starting msgr at 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 librados: starting objecter 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 Processor -- start 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 -- start start 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 librados: setting wanted keys 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 1 librados: calling monclient init 2024-07-27T15:18:00.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.355+0000 7fa9bed9e640 10 monclient: init 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9b8072b70 0x7fa9b814bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b8072b70 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b80af970 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b80b1c80 0x7fa9b813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9b80b1c80 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b80b2520 con 0x7fa9b8072b70 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b80b23a0 con 0x7fa9b80af970 2024-07-27T15:18:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b814d510 con 0x7fa9b80b1c80 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bed9e640 10 monclient(hunting): authenticate will time out at 3420.382151s 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bcb3a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9b8072b70 0x7fa9b814bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9bd33b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b80b1c80 0x7fa9b813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.87:45584/0 (socket says 172.21.15.87:45584) 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 1 -- 172.21.15.87:0/2506660511 learned_addr learned my addr 172.21.15.87:0/2506660511 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 10 monclient(hunting): get_auth_request con 0x7fa9b80af970 auth_method 0 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.356+0000 7fa9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): get_auth_request con 0x7fa9b80b1c80 auth_method 0 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): get_auth_request con 0x7fa9b8072b70 auth_method 0 2024-07-27T15:18:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bd33b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.357+0000 7fa9bcb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 10 monclient(hunting): handle_auth_done global_id 21979 payload 824 2024-07-27T15:18:00.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 -- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b80b1c80 msgr2=0x7fa9b813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b80b1c80 0x7fa9b813ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 -- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9b8072b70 msgr2=0x7fa9b814bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9b8072b70 0x7fa9b814bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 monclient: found mon.noname-b 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.360+0000 7fa9b7fff640 1 -- 172.21.15.87:0/2506660511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b80aad60 con 0x7fa9b80af970 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.362+0000 7fa9b7fff640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e980 tx=0x7fa9a400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.362+0000 7fa9b57fa640 10 client.?.objecter ms_handle_connect 0x7fa9b80af970 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.363+0000 7fa9b57fa640 10 client.?.objecter resend_mon_ops 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4017070 con 0x7fa9b80af970 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: dump: 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a400ca00 con 0x7fa9b80af970 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 5 monclient: authenticate success, global_id 21979 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 10 monclient: _renew_subs 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b80ab040 con 0x7fa9b80af970 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400cdd0 con 0x7fa9b80af970 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b57fa640 10 monclient: dump: 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.372 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9b6ffd640 4 set_mon_vals no callback set 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 10 client.21979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 10 monclient: _renew_subs 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b813f490 con 0x7fa9b80af970 2024-07-27T15:18:00.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.367+0000 7fa9bed9e640 1 librados: init done 2024-07-27T15:18:00.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.368+0000 7fa9bed9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.368+0000 7fa9bed9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.368+0000 7fa9bed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.368+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b80b4ca0 con 0x7fa9b80af970 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.631+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa9a401a070 con 0x7fa9b80af970 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.632+0000 7fa9b57fa640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 0x7fa99c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.632+0000 7fa9bcb3a640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 0x7fa99c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.632+0000 7fa9bcb3a640 10 monclient: get_auth_request con 0x7fa99c06d7c0 auth_method 0 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.633+0000 7fa9b57fa640 10 client.21979.objecter ms_handle_connect 0x7fa99c06d7c0 2024-07-27T15:18:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.633+0000 7fa9bcb3a640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 0x7fa99c06fc80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac036000 tx=0x7fa9ac03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.636+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7fa9a40579b0 con 0x7fa9b80af970 2024-07-27T15:18:00.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.636+0000 7fa9b57fa640 10 client.21979.objecter ms_dispatch 0x7fa9b8000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:18:00.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.636+0000 7fa9b57fa640 3 client.21979.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:18:00.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.636+0000 7fa9b57fa640 3 client.21979.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:18:00.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.636+0000 7fa9b57fa640 20 client.21979.objecter dump_active .. 0 homeless 2024-07-27T15:18:00.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.646+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7fa9a4059910 con 0x7fa9b80af970 2024-07-27T15:18:00.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.646+0000 7fa9b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.646+0000 7fa9b57fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:18:00.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.827+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fa9b814ba20 con 0x7fa99c06d7c0 2024-07-27T15:18:00.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.828+0000 7fa9b57fa640 1 -- 172.21.15.87:0/2506660511 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa9b814ba20 con 0x7fa99c06d7c0 2024-07-27T15:18:00.829 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 7 to deep-scrub 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.829+0000 7fa9bed9e640 10 librados: watch_flush enter 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.829+0000 7fa9bed9e640 10 librados: watch_flush exit 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.829+0000 7fa9bed9e640 20 client.21979.objecter shutdown clearing up homeless session... 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.829+0000 7fa9bed9e640 10 client.21979.objecter successfully canceled tick 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 msgr2=0x7fa99c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 0x7fa99c06fc80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac036000 tx=0x7fa9ac03c040 comp rx=0 tx=0).stop 2024-07-27T15:18:00.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 10 monclient: shutdown 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 msgr2=0x7fa9b814b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e980 tx=0x7fa9a400ee40 comp rx=0 tx=0).stop 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 shutdown_connections 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7fa99c06d7c0 0x7fa99c06fc80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9b80b1c80 0x7fa9b813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b80af970 0x7fa9b814b270 unknown :-1 s=CLOSED pgs=5052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 --2- 172.21.15.87:0/2506660511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9b8072b70 0x7fa9b814bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 >> 172.21.15.87:0/2506660511 conn(0x7fa9b8068150 msgr2=0x7fa9b813fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 shutdown_connections 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 -- 172.21.15.87:0/2506660511 wait complete. 2024-07-27T15:18:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.830+0000 7fa9bed9e640 1 librados: shutdown 2024-07-27T15:18:00.842 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.940+0000 7f4da0467640 10 monclient: get_monmap_and_config 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.940+0000 7f4da0467640 10 monclient: build_initial_monmap 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.940+0000 7f4da0467640 10 monclient: monmap: 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:18:00.941655+0000 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:18:00.941655+0000 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:00.942 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.941+0000 7f4da0467640 1 Processor -- start 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.941+0000 7f4da0467640 1 -- start start 2024-07-27T15:18:00.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.941+0000 7f4da0467640 10 monclient: init 2024-07-27T15:18:00.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.941+0000 7f4da0467640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.941+0000 7f4da0467640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 0x7f4d98146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-b con 0x7f4d980678a0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4d980b0420 0x7f4d9813fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-c con 0x7f4d980b0420 addr v2:172.21.15.112:3300/0 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d980b0e60 addr v1:172.21.15.112:6789/0 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d980736d0 con 0x7f4d980b0e60 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f4d98067c80 con 0x7f4d980b0420 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f4d980b2080 con 0x7f4d980678a0 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4da0467640 10 monclient(hunting): authenticate will time out at 3420.968083s 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4d9e203640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 0x7f4d98146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.942+0000 7f4d9da02640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4d980b0420 0x7f4d9813fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9ea04640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4d980b0e60 0x7f4d98142b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.87:52974/0 (socket says 172.21.15.87:52974) 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9ea04640 1 -- 172.21.15.87:0/147803870 learned_addr learned my addr 172.21.15.87:0/147803870 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): get_auth_request con 0x7f4d980678a0 auth_method 0 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9da02640 10 monclient(hunting): get_auth_request con 0x7f4d980b0420 auth_method 0 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9da02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9da02640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.943+0000 7f4d9da02640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.944+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.944+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.944+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 10 monclient(hunting): handle_auth_done global_id 15578 payload 274 2024-07-27T15:18:00.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 1 -- 172.21.15.87:0/147803870 >> v2:172.21.15.112:3300/0 conn(0x7f4d980b0420 msgr2=0x7f4d9813fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:00.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 1 --2- 172.21.15.87:0/147803870 >> v2:172.21.15.112:3300/0 conn(0x7f4d980b0420 0x7f4d9813fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 1 -- 172.21.15.87:0/147803870 >> v1:172.21.15.112:6789/0 conn(0x7f4d980b0e60 legacy=0x7f4d98142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 1 monclient: found mon.noname-b 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.945+0000 7f4d9e203640 1 -- 172.21.15.87:0/147803870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d980b0940 con 0x7f4d980678a0 2024-07-27T15:18:00.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.947+0000 7f4d9e203640 1 --2- 172.21.15.87:0/147803870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 0x7f4d98146080 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f4d880097a0 tx=0x7f4d8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2a66f014f0efc5 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 1 -- 172.21.15.87:0/147803870 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88035020 con 0x7f4d980678a0 2024-07-27T15:18:00.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:00.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: dump: 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 1 -- 172.21.15.87:0/147803870 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d88031830 con 0x7f4d980678a0 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4da0467640 5 monclient: authenticate success, global_id 15578 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4da0467640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 1 -- 172.21.15.87:0/147803870 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88031c00 con 0x7f4d980678a0 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4da0467640 10 monclient: get_monmap_and_config success 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4da0467640 4 set_mon_vals no callback set 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.950+0000 7f4d9d201640 10 monclient: dump: 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.955 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 10 monclient: shutdown 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 -- 172.21.15.87:0/147803870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 msgr2=0x7f4d98146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 --2- 172.21.15.87:0/147803870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 0x7f4d98146080 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f4d880097a0 tx=0x7f4d8802b8d0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 -- 172.21.15.87:0/147803870 shutdown_connections 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 --2- 172.21.15.87:0/147803870 >> v2:172.21.15.112:3300/0 conn(0x7f4d980b0420 0x7f4d9813fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 --2- 172.21.15.87:0/147803870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980678a0 0x7f4d98146080 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 -- 172.21.15.87:0/147803870 >> 172.21.15.87:0/147803870 conn(0x7f4d980b6260 msgr2=0x7f4d9806b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 -- 172.21.15.87:0/147803870 shutdown_connections 2024-07-27T15:18:00.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.951+0000 7f4da0467640 1 -- 172.21.15.87:0/147803870 wait complete. 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.952+0000 7f4da0467640 10 monclient: build_initial_monmap 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.952+0000 7f4da0467640 10 monclient: monmap: 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 librados: starting msgr at 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 librados: starting objecter 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 Processor -- start 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 -- start start 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 librados: setting wanted keys 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 librados: calling monclient init 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient: init 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 0x7f4d981cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-b con 0x7f4d980678a0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 0x7f4d981cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-c con 0x7f4d980b0420 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:00.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d980b0e60 0x7f4d981d8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): picked mon.noname-a con 0x7f4d980b0e60 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4d98146740 con 0x7f4d980b0e60 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4d981465c0 con 0x7f4d980678a0 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f4d981468c0 con 0x7f4d980b0420 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.953+0000 7f4da0467640 10 monclient(hunting): authenticate will time out at 3420.979755s 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 0x7f4d981cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9ea04640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d980b0e60 0x7f4d981d8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9e203640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 0x7f4d981cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:00.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 0x7f4d981cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37344/0 (socket says 172.21.15.87:37344) 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 1 -- 172.21.15.87:0/2222637414 learned_addr learned my addr 172.21.15.87:0/2222637414 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 10 monclient(hunting): get_auth_request con 0x7f4d980b0420 auth_method 0 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9da02640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9e203640 10 monclient(hunting): get_auth_request con 0x7f4d980678a0 auth_method 0 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9e203640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9e203640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9e203640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9ea04640 10 monclient(hunting): get_auth_request con 0x7f4d980b0e60 auth_method 0 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9ea04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9ea04640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:00.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.954+0000 7f4d9ea04640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9da02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9e203640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9ea04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9ea04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.955+0000 7f4d9ea04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.957+0000 7f4d9e203640 10 monclient(hunting): handle_auth_done global_id 21988 payload 824 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.957+0000 7f4d9e203640 1 -- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 msgr2=0x7f4d981cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.957+0000 7f4d9e203640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 0x7f4d981cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 1 -- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d980b0e60 msgr2=0x7f4d981d8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:00.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d980b0e60 0x7f4d981d8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:00.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 10 monclient: _finish_hunting 0 2024-07-27T15:18:00.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 1 monclient: found mon.noname-b 2024-07-27T15:18:00.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:00.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.958+0000 7f4d9e203640 1 -- 172.21.15.87:0/2222637414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d981411e0 con 0x7f4d980678a0 2024-07-27T15:18:00.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.960+0000 7f4d9e203640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 0x7f4d981cd720 secure :-1 s=READY pgs=5054 cs=0 l=1 rev1=1 crypto rx=0x7f4d88033000 tx=0x7f4d880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:00.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.960+0000 7f4d8effd640 10 client.?.objecter ms_handle_connect 0x7f4d980678a0 2024-07-27T15:18:00.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.960+0000 7f4d8effd640 10 client.?.objecter resend_mon_ops 2024-07-27T15:18:00.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.963+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88041070 con 0x7f4d980678a0 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.963+0000 7f4d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.963+0000 7f4d8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.963+0000 7f4d8effd640 10 monclient: dump: 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: _finish_auth 0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: _check_auth_tickets 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8803b920 con 0x7f4d980678a0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 5 monclient: authenticate success, global_id 21988 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 10 monclient: _renew_subs 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d981ce410 con 0x7f4d980678a0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88035040 con 0x7f4d980678a0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d8effd640 10 monclient: dump: 2024-07-27T15:18:00.968 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4d9ca00640 4 set_mon_vals no callback set 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 10 client.21988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 10 monclient: _renew_subs 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d981452e0 con 0x7f4d980678a0 2024-07-27T15:18:00.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.964+0000 7f4da0467640 1 librados: init done 2024-07-27T15:18:00.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.965+0000 7f4da0467640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.965+0000 7f4da0467640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:00.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.965+0000 7f4da0467640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.112:3300/0 2024-07-27T15:18:00.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:00.965+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d98140540 con 0x7f4d980678a0 2024-07-27T15:18:01.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.221+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4d8804c060 con 0x7f4d980678a0 2024-07-27T15:18:01.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.221+0000 7f4d8effd640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 0x7f4d7c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.222+0000 7f4d9da02640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 0x7f4d7c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.222+0000 7f4d9da02640 10 monclient: get_auth_request con 0x7f4d7c06d810 auth_method 0 2024-07-27T15:18:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.222+0000 7f4d8effd640 10 client.21988.objecter ms_handle_connect 0x7f4d7c06d810 2024-07-27T15:18:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.222+0000 7f4d9da02640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 0x7f4d7c06fcd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4d981cf0d0 tx=0x7f4d94009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.226+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f4d88078dc0 con 0x7f4d980678a0 2024-07-27T15:18:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.226+0000 7f4d8effd640 10 client.21988.objecter ms_dispatch 0x7f4d98000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:18:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.226+0000 7f4d8effd640 3 client.21988.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:18:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.226+0000 7f4d8effd640 3 client.21988.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:18:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.226+0000 7f4d8effd640 20 client.21988.objecter dump_active .. 0 homeless 2024-07-27T15:18:01.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.240+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f4d981452e0 con 0x7f4d980678a0 2024-07-27T15:18:01.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.240+0000 7f4d8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:01.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.240+0000 7f4d8effd640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:18:01.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.432+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d980af890 con 0x7f4d7c06d810 2024-07-27T15:18:01.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.436+0000 7f4d8effd640 1 -- 172.21.15.87:0/2222637414 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+115876 (secure 0 0 0) 0x7f4d980af890 con 0x7f4d7c06d810 2024-07-27T15:18:01.438 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:18:01.439 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 10 librados: watch_flush enter 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 10 librados: watch_flush exit 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 20 client.21988.objecter shutdown clearing up homeless session... 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 10 client.21988.objecter successfully canceled tick 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 msgr2=0x7f4d7c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 0x7f4d7c06fcd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4d981cf0d0 tx=0x7f4d94009190 comp rx=0 tx=0).stop 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 10 monclient: shutdown 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:01.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 msgr2=0x7f4d981cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 0x7f4d981cd720 secure :-1 s=READY pgs=5054 cs=0 l=1 rev1=1 crypto rx=0x7f4d88033000 tx=0x7f4d880046c0 comp rx=0 tx=0).stop 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 shutdown_connections 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f4d7c06d810 0x7f4d7c06fcd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d980b0e60 0x7f4d981d8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d980b0420 0x7f4d981cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 --2- 172.21.15.87:0/2222637414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4d980678a0 0x7f4d981cd720 unknown :-1 s=CLOSED pgs=5054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 >> 172.21.15.87:0/2222637414 conn(0x7f4d980b6260 msgr2=0x7f4d98139c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 shutdown_connections 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 -- 172.21.15.87:0/2222637414 wait complete. 2024-07-27T15:18:01.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:01.439+0000 7f4da0467640 1 librados: shutdown 2024-07-27T15:18:01.452 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":2582,"stamp":"2024-07-27T15:18:00.495310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":122,"num_write_kb":1381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595288,"kb_used_data":44608,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754379432,"statfs":{"total":773094113280,"available":772484538368,"internally_reserved":0,"allocated":45678592,"data_stored":22072672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563793456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.156250"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":2818,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.582358+0000","last_change":"2024-07-27T14:42:41.580179+0000","last_active":"2024-07-27T15:17:50.582358+0000","last_peered":"2024-07-27T15:17:50.582358+0000","last_clean":"2024-07-27T15:17:50.582358+0000","last_became_active":"2024-07-27T14:42:41.574286+0000","last_became_peered":"2024-07-27T14:42:41.574286+0000","last_unstale":"2024-07-27T15:17:50.582358+0000","last_undegraded":"2024-07-27T15:17:50.582358+0000","last_fullsized":"2024-07-27T15:17:50.582358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:03:00.545391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":2827,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.622236+0000","last_change":"2024-07-27T14:42:33.146971+0000","last_active":"2024-07-27T15:17:50.622236+0000","last_peered":"2024-07-27T15:17:50.622236+0000","last_clean":"2024-07-27T15:17:50.622236+0000","last_became_active":"2024-07-27T14:42:33.130123+0000","last_became_peered":"2024-07-27T14:42:33.130123+0000","last_unstale":"2024-07-27T15:17:50.622236+0000","last_undegraded":"2024-07-27T15:17:50.622236+0000","last_fullsized":"2024-07-27T15:17:50.622236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.380549+0000","last_change":"2024-07-27T14:42:32.205230+0000","last_active":"2024-07-27T15:17:49.380549+0000","last_peered":"2024-07-27T15:17:49.380549+0000","last_clean":"2024-07-27T15:17:49.380549+0000","last_became_active":"2024-07-27T14:42:32.196113+0000","last_became_peered":"2024-07-27T14:42:32.196113+0000","last_unstale":"2024-07-27T15:17:49.380549+0000","last_undegraded":"2024-07-27T15:17:49.380549+0000","last_fullsized":"2024-07-27T15:17:49.380549+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:11:27.161409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2831,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.377590+0000","last_change":"2024-07-27T14:42:33.556536+0000","last_active":"2024-07-27T15:17:49.377590+0000","last_peered":"2024-07-27T15:17:49.377590+0000","last_clean":"2024-07-27T15:17:49.377590+0000","last_became_active":"2024-07-27T14:42:33.550008+0000","last_became_peered":"2024-07-27T14:42:33.550008+0000","last_unstale":"2024-07-27T15:17:49.377590+0000","last_undegraded":"2024-07-27T15:17:49.377590+0000","last_fullsized":"2024-07-27T15:17:49.377590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:19:45.505942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.581470+0000","last_change":"2024-07-27T14:42:33.671344+0000","last_active":"2024-07-27T15:17:50.581470+0000","last_peered":"2024-07-27T15:17:50.581470+0000","last_clean":"2024-07-27T15:17:50.581470+0000","last_became_active":"2024-07-27T14:42:33.554937+0000","last_became_peered":"2024-07-27T14:42:33.554937+0000","last_unstale":"2024-07-27T15:17:50.581470+0000","last_undegraded":"2024-07-27T15:17:50.581470+0000","last_fullsized":"2024-07-27T15:17:50.581470+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2813,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.341058+0000","last_change":"2024-07-27T15:17:57.341860+0000","last_active":"2024-07-27T15:17:57.341058+0000","last_peered":"2024-07-27T15:17:57.341058+0000","last_clean":"2024-07-27T15:17:57.341058+0000","last_became_active":"2024-07-27T14:42:43.877378+0000","last_became_peered":"2024-07-27T14:42:43.877378+0000","last_unstale":"2024-07-27T15:17:57.341058+0000","last_undegraded":"2024-07-27T15:17:57.341058+0000","last_fullsized":"2024-07-27T15:17:57.341058+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.339321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.339321+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.339321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:37:21.811853+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":2818,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.369954+0000","last_change":"2024-07-27T14:42:35.229319+0000","last_active":"2024-07-27T15:17:49.369954+0000","last_peered":"2024-07-27T15:17:49.369954+0000","last_clean":"2024-07-27T15:17:49.369954+0000","last_became_active":"2024-07-27T14:42:35.220195+0000","last_became_peered":"2024-07-27T14:42:35.220195+0000","last_unstale":"2024-07-27T15:17:49.369954+0000","last_undegraded":"2024-07-27T15:17:49.369954+0000","last_fullsized":"2024-07-27T15:17:49.369954+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2834,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.362335+0000","last_change":"2024-07-27T14:42:33.873297+0000","last_active":"2024-07-27T15:17:49.362335+0000","last_peered":"2024-07-27T15:17:49.362335+0000","last_clean":"2024-07-27T15:17:49.362335+0000","last_became_active":"2024-07-27T14:42:33.867602+0000","last_became_peered":"2024-07-27T14:42:33.867602+0000","last_unstale":"2024-07-27T15:17:49.362335+0000","last_undegraded":"2024-07-27T15:17:49.362335+0000","last_fullsized":"2024-07-27T15:17:49.362335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:17:09.550687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":2836,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:55.588628+0000","last_change":"2024-07-27T15:17:55.589452+0000","last_active":"2024-07-27T15:17:55.588628+0000","last_peered":"2024-07-27T15:17:55.588628+0000","last_clean":"2024-07-27T15:17:55.588628+0000","last_became_active":"2024-07-27T14:42:34.746905+0000","last_became_peered":"2024-07-27T14:42:34.746905+0000","last_unstale":"2024-07-27T15:17:55.588628+0000","last_undegraded":"2024-07-27T15:17:55.588628+0000","last_fullsized":"2024-07-27T15:17:55.588628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:55.587059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:55.587059+0000","last_clean_scrub_stamp":"2024-07-27T15:17:55.587059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:03.467232+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":2829,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.617904+0000","last_change":"2024-07-27T14:42:37.296338+0000","last_active":"2024-07-27T15:17:50.617904+0000","last_peered":"2024-07-27T15:17:50.617904+0000","last_clean":"2024-07-27T15:17:50.617904+0000","last_became_active":"2024-07-27T14:42:37.287685+0000","last_became_peered":"2024-07-27T14:42:37.287685+0000","last_unstale":"2024-07-27T15:17:50.617904+0000","last_undegraded":"2024-07-27T15:17:50.617904+0000","last_fullsized":"2024-07-27T15:17:50.617904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.362421+0000","last_change":"2024-07-27T14:42:35.360457+0000","last_active":"2024-07-27T15:17:49.362421+0000","last_peered":"2024-07-27T15:17:49.362421+0000","last_clean":"2024-07-27T15:17:49.362421+0000","last_became_active":"2024-07-27T14:42:35.354552+0000","last_became_peered":"2024-07-27T14:42:35.354552+0000","last_unstale":"2024-07-27T15:17:49.362421+0000","last_undegraded":"2024-07-27T15:17:49.362421+0000","last_fullsized":"2024-07-27T15:17:49.362421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2806,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.332118+0000","last_change":"2024-07-27T14:42:34.577783+0000","last_active":"2024-07-27T15:17:50.332118+0000","last_peered":"2024-07-27T15:17:50.332118+0000","last_clean":"2024-07-27T15:17:50.332118+0000","last_became_active":"2024-07-27T14:42:34.568719+0000","last_became_peered":"2024-07-27T14:42:34.568719+0000","last_unstale":"2024-07-27T15:17:50.332118+0000","last_undegraded":"2024-07-27T15:17:50.332118+0000","last_fullsized":"2024-07-27T15:17:50.332118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:14:22.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1468'78","reported_seq":2942,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:56.423173+0000","last_change":"2024-07-27T15:17:56.424691+0000","last_active":"2024-07-27T15:17:56.423173+0000","last_peered":"2024-07-27T15:17:56.423173+0000","last_clean":"2024-07-27T15:17:56.423173+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T15:17:56.423173+0000","last_undegraded":"2024-07-27T15:17:56.423173+0000","last_fullsized":"2024-07-27T15:17:56.423173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'78","last_scrub_stamp":"2024-07-27T15:17:56.420345+0000","last_deep_scrub":"1468'78","last_deep_scrub_stamp":"2024-07-27T15:17:56.420345+0000","last_clean_scrub_stamp":"2024-07-27T15:17:56.420345+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:46:32.650722+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":2852,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.610385+0000","last_change":"2024-07-27T14:41:21.982374+0000","last_active":"2024-07-27T15:17:50.610385+0000","last_peered":"2024-07-27T15:17:50.610385+0000","last_clean":"2024-07-27T15:17:50.610385+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T15:17:50.610385+0000","last_undegraded":"2024-07-27T15:17:50.610385+0000","last_fullsized":"2024-07-27T15:17:50.610385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":2830,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.624628+0000","last_change":"2024-07-27T14:42:37.311409+0000","last_active":"2024-07-27T15:17:50.624628+0000","last_peered":"2024-07-27T15:17:50.624628+0000","last_clean":"2024-07-27T15:17:50.624628+0000","last_became_active":"2024-07-27T14:42:37.189428+0000","last_became_peered":"2024-07-27T14:42:37.189428+0000","last_unstale":"2024-07-27T15:17:50.624628+0000","last_undegraded":"2024-07-27T15:17:50.624628+0000","last_fullsized":"2024-07-27T15:17:50.624628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2859,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.583693+0000","last_change":"2024-07-27T15:17:57.584731+0000","last_active":"2024-07-27T15:17:57.583693+0000","last_peered":"2024-07-27T15:17:57.583693+0000","last_clean":"2024-07-27T15:17:57.583693+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T15:17:57.583693+0000","last_undegraded":"2024-07-27T15:17:57.583693+0000","last_fullsized":"2024-07-27T15:17:57.583693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.582075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.582075+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.582075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:18:38.448714+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2834,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.382451+0000","last_change":"2024-07-27T14:42:32.948619+0000","last_active":"2024-07-27T15:17:49.382451+0000","last_peered":"2024-07-27T15:17:49.382451+0000","last_clean":"2024-07-27T15:17:49.382451+0000","last_became_active":"2024-07-27T14:42:32.825779+0000","last_became_peered":"2024-07-27T14:42:32.825779+0000","last_unstale":"2024-07-27T15:17:49.382451+0000","last_undegraded":"2024-07-27T15:17:49.382451+0000","last_fullsized":"2024-07-27T15:17:49.382451+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:44:30.813522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2827,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.321810+0000","last_change":"2024-07-27T14:41:21.968387+0000","last_active":"2024-07-27T15:17:50.321810+0000","last_peered":"2024-07-27T15:17:50.321810+0000","last_clean":"2024-07-27T15:17:50.321810+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T15:17:50.321810+0000","last_undegraded":"2024-07-27T15:17:50.321810+0000","last_fullsized":"2024-07-27T15:17:50.321810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:59:37.205221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2817,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.590866+0000","last_change":"2024-07-27T14:43:03.368313+0000","last_active":"2024-07-27T15:17:50.590866+0000","last_peered":"2024-07-27T15:17:50.590866+0000","last_clean":"2024-07-27T15:17:50.590866+0000","last_became_active":"2024-07-27T14:43:03.360033+0000","last_became_peered":"2024-07-27T14:43:03.360033+0000","last_unstale":"2024-07-27T15:17:50.590866+0000","last_undegraded":"2024-07-27T15:17:50.590866+0000","last_fullsized":"2024-07-27T15:17:50.590866+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:21:29.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":2853,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.601096+0000","last_change":"2024-07-27T14:41:21.962086+0000","last_active":"2024-07-27T15:17:50.601096+0000","last_peered":"2024-07-27T15:17:50.601096+0000","last_clean":"2024-07-27T15:17:50.601096+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T15:17:50.601096+0000","last_undegraded":"2024-07-27T15:17:50.601096+0000","last_fullsized":"2024-07-27T15:17:50.601096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2819,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.580449+0000","last_change":"2024-07-27T14:42:33.420873+0000","last_active":"2024-07-27T15:17:50.580449+0000","last_peered":"2024-07-27T15:17:50.580449+0000","last_clean":"2024-07-27T15:17:50.580449+0000","last_became_active":"2024-07-27T14:42:33.415149+0000","last_became_peered":"2024-07-27T14:42:33.415149+0000","last_unstale":"2024-07-27T15:17:50.580449+0000","last_undegraded":"2024-07-27T15:17:50.580449+0000","last_fullsized":"2024-07-27T15:17:50.580449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:45:18.005668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2854,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.371119+0000","last_change":"2024-07-27T14:41:21.836729+0000","last_active":"2024-07-27T15:17:49.371119+0000","last_peered":"2024-07-27T15:17:49.371119+0000","last_clean":"2024-07-27T15:17:49.371119+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T15:17:49.371119+0000","last_undegraded":"2024-07-27T15:17:49.371119+0000","last_fullsized":"2024-07-27T15:17:49.371119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:49:24.494551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2820,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.588719+0000","last_change":"2024-07-27T14:42:57.118705+0000","last_active":"2024-07-27T15:17:50.588719+0000","last_peered":"2024-07-27T15:17:50.588719+0000","last_clean":"2024-07-27T15:17:50.588719+0000","last_became_active":"2024-07-27T14:42:57.112921+0000","last_became_peered":"2024-07-27T14:42:57.112921+0000","last_unstale":"2024-07-27T15:17:50.588719+0000","last_undegraded":"2024-07-27T15:17:50.588719+0000","last_fullsized":"2024-07-27T15:17:50.588719+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:02:08.809341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2848,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.627000+0000","last_change":"2024-07-27T14:41:21.974198+0000","last_active":"2024-07-27T15:17:50.627000+0000","last_peered":"2024-07-27T15:17:50.627000+0000","last_clean":"2024-07-27T15:17:50.627000+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T15:17:50.627000+0000","last_undegraded":"2024-07-27T15:17:50.627000+0000","last_fullsized":"2024-07-27T15:17:50.627000+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2819,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.578450+0000","last_change":"2024-07-27T14:42:36.978696+0000","last_active":"2024-07-27T15:17:50.578450+0000","last_peered":"2024-07-27T15:17:50.578450+0000","last_clean":"2024-07-27T15:17:50.578450+0000","last_became_active":"2024-07-27T14:42:36.941428+0000","last_became_peered":"2024-07-27T14:42:36.941428+0000","last_unstale":"2024-07-27T15:17:50.578450+0000","last_undegraded":"2024-07-27T15:17:50.578450+0000","last_fullsized":"2024-07-27T15:17:50.578450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:34:55.684245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2841,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.392531+0000","last_change":"2024-07-27T14:41:21.974223+0000","last_active":"2024-07-27T15:17:49.392531+0000","last_peered":"2024-07-27T15:17:49.392531+0000","last_clean":"2024-07-27T15:17:49.392531+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T15:17:49.392531+0000","last_undegraded":"2024-07-27T15:17:49.392531+0000","last_fullsized":"2024-07-27T15:17:49.392531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:14:40.662818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2825,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.362348+0000","last_change":"2024-07-27T14:42:32.399654+0000","last_active":"2024-07-27T15:17:49.362348+0000","last_peered":"2024-07-27T15:17:49.362348+0000","last_clean":"2024-07-27T15:17:49.362348+0000","last_became_active":"2024-07-27T14:42:32.329977+0000","last_became_peered":"2024-07-27T14:42:32.329977+0000","last_unstale":"2024-07-27T15:17:49.362348+0000","last_undegraded":"2024-07-27T15:17:49.362348+0000","last_fullsized":"2024-07-27T15:17:49.362348+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:17:59.414451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2845,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.637740+0000","last_change":"2024-07-27T14:41:21.970896+0000","last_active":"2024-07-27T15:17:50.637740+0000","last_peered":"2024-07-27T15:17:50.637740+0000","last_clean":"2024-07-27T15:17:50.637740+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T15:17:50.637740+0000","last_undegraded":"2024-07-27T15:17:50.637740+0000","last_fullsized":"2024-07-27T15:17:50.637740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:41:16.982882+0000","last_clean_scrub_stamp":"2024-07-27T14:41:16.982882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1468'8","reported_seq":2842,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:59.061049+0000","last_change":"2024-07-27T15:17:59.062048+0000","last_active":"2024-07-27T15:17:59.061049+0000","last_peered":"2024-07-27T15:17:59.061049+0000","last_clean":"2024-07-27T15:17:59.061049+0000","last_became_active":"2024-07-27T14:42:34.791352+0000","last_became_peered":"2024-07-27T14:42:34.791352+0000","last_unstale":"2024-07-27T15:17:59.061049+0000","last_undegraded":"2024-07-27T15:17:59.061049+0000","last_fullsized":"2024-07-27T15:17:59.061049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'8","last_scrub_stamp":"2024-07-27T15:17:59.059180+0000","last_deep_scrub":"1468'8","last_deep_scrub_stamp":"2024-07-27T15:17:59.059180+0000","last_clean_scrub_stamp":"2024-07-27T15:17:59.059180+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:58:03.698313+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":2817,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.359524+0000","last_change":"2024-07-27T14:42:35.255013+0000","last_active":"2024-07-27T15:17:49.359524+0000","last_peered":"2024-07-27T15:17:49.359524+0000","last_clean":"2024-07-27T15:17:49.359524+0000","last_became_active":"2024-07-27T14:42:35.248761+0000","last_became_peered":"2024-07-27T14:42:35.248761+0000","last_unstale":"2024-07-27T15:17:49.359524+0000","last_undegraded":"2024-07-27T15:17:49.359524+0000","last_fullsized":"2024-07-27T15:17:49.359524+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2823,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.373104+0000","last_change":"2024-07-27T14:42:53.137283+0000","last_active":"2024-07-27T15:17:49.373104+0000","last_peered":"2024-07-27T15:17:49.373104+0000","last_clean":"2024-07-27T15:17:49.373104+0000","last_became_active":"2024-07-27T14:42:53.131434+0000","last_became_peered":"2024-07-27T14:42:53.131434+0000","last_unstale":"2024-07-27T15:17:49.373104+0000","last_undegraded":"2024-07-27T15:17:49.373104+0000","last_fullsized":"2024-07-27T15:17:49.373104+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:47.643598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2823,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.366253+0000","last_change":"2024-07-27T14:42:32.763809+0000","last_active":"2024-07-27T15:17:49.366253+0000","last_peered":"2024-07-27T15:17:49.366253+0000","last_clean":"2024-07-27T15:17:49.366253+0000","last_became_active":"2024-07-27T14:42:32.755122+0000","last_became_peered":"2024-07-27T14:42:32.755122+0000","last_unstale":"2024-07-27T15:17:49.366253+0000","last_undegraded":"2024-07-27T15:17:49.366253+0000","last_fullsized":"2024-07-27T15:17:49.366253+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:18:44.350815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2807,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.314357+0000","last_change":"2024-07-27T14:42:34.656806+0000","last_active":"2024-07-27T15:17:50.314357+0000","last_peered":"2024-07-27T15:17:50.314357+0000","last_clean":"2024-07-27T15:17:50.314357+0000","last_became_active":"2024-07-27T14:42:34.647534+0000","last_became_peered":"2024-07-27T14:42:34.647534+0000","last_unstale":"2024-07-27T15:17:50.314357+0000","last_undegraded":"2024-07-27T15:17:50.314357+0000","last_fullsized":"2024-07-27T15:17:50.314357+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:43:30.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":2829,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.635375+0000","last_change":"2024-07-27T14:42:37.982514+0000","last_active":"2024-07-27T15:17:50.635375+0000","last_peered":"2024-07-27T15:17:50.635375+0000","last_clean":"2024-07-27T15:17:50.635375+0000","last_became_active":"2024-07-27T14:42:37.973009+0000","last_became_peered":"2024-07-27T14:42:37.973009+0000","last_unstale":"2024-07-27T15:17:50.635375+0000","last_undegraded":"2024-07-27T15:17:50.635375+0000","last_fullsized":"2024-07-27T15:17:50.635375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.394466+0000","last_change":"2024-07-27T14:42:32.206159+0000","last_active":"2024-07-27T15:17:49.394466+0000","last_peered":"2024-07-27T15:17:49.394466+0000","last_clean":"2024-07-27T15:17:49.394466+0000","last_became_active":"2024-07-27T14:42:32.194637+0000","last_became_peered":"2024-07-27T14:42:32.194637+0000","last_unstale":"2024-07-27T15:17:49.394466+0000","last_undegraded":"2024-07-27T15:17:49.394466+0000","last_fullsized":"2024-07-27T15:17:49.394466+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:04:59.701883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2806,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.328362+0000","last_change":"2024-07-27T14:42:30.522614+0000","last_active":"2024-07-27T15:17:50.328362+0000","last_peered":"2024-07-27T15:17:50.328362+0000","last_clean":"2024-07-27T15:17:50.328362+0000","last_became_active":"2024-07-27T14:42:30.517011+0000","last_became_peered":"2024-07-27T14:42:30.517011+0000","last_unstale":"2024-07-27T15:17:50.328362+0000","last_undegraded":"2024-07-27T15:17:50.328362+0000","last_fullsized":"2024-07-27T15:17:50.328362+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:36:37.080542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2821,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.384039+0000","last_change":"2024-07-27T14:42:44.953045+0000","last_active":"2024-07-27T15:17:49.384039+0000","last_peered":"2024-07-27T15:17:49.384039+0000","last_clean":"2024-07-27T15:17:49.384039+0000","last_became_active":"2024-07-27T14:42:44.426410+0000","last_became_peered":"2024-07-27T14:42:44.426410+0000","last_unstale":"2024-07-27T15:17:49.384039+0000","last_undegraded":"2024-07-27T15:17:49.384039+0000","last_fullsized":"2024-07-27T15:17:49.384039+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:38:49.627031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":2816,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:49.357717+0000","last_change":"2024-07-27T14:43:09.714646+0000","last_active":"2024-07-27T15:17:49.357717+0000","last_peered":"2024-07-27T15:17:49.357717+0000","last_clean":"2024-07-27T15:17:49.357717+0000","last_became_active":"2024-07-27T14:43:09.707672+0000","last_became_peered":"2024-07-27T14:43:09.707672+0000","last_unstale":"2024-07-27T15:17:49.357717+0000","last_undegraded":"2024-07-27T15:17:49.357717+0000","last_fullsized":"2024-07-27T15:17:49.357717+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.080606+0000","last_change":"2024-07-27T15:17:57.081377+0000","last_active":"2024-07-27T15:17:57.080606+0000","last_peered":"2024-07-27T15:17:57.080606+0000","last_clean":"2024-07-27T15:17:57.080606+0000","last_became_active":"2024-07-27T14:42:32.384600+0000","last_became_peered":"2024-07-27T14:42:32.384600+0000","last_unstale":"2024-07-27T15:17:57.080606+0000","last_undegraded":"2024-07-27T15:17:57.080606+0000","last_fullsized":"2024-07-27T15:17:57.080606+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.079054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.079054+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.079054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:37:13.227424+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2806,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:50.319942+0000","last_change":"2024-07-27T14:42:33.230973+0000","last_active":"2024-07-27T15:17:50.319942+0000","last_peered":"2024-07-27T15:17:50.319942+0000","last_clean":"2024-07-27T15:17:50.319942+0000","last_became_active":"2024-07-27T14:42:33.224999+0000","last_became_peered":"2024-07-27T14:42:33.224999+0000","last_unstale":"2024-07-27T15:17:50.319942+0000","last_undegraded":"2024-07-27T15:17:50.319942+0000","last_fullsized":"2024-07-27T15:17:50.319942+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T14:42:27.068054+0000","last_clean_scrub_stamp":"2024-07-27T14:42:27.068054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T23:37:06.846510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:58.066453+0000","last_change":"2024-07-27T15:17:58.067244+0000","last_active":"2024-07-27T15:17:58.066453+0000","last_peered":"2024-07-27T15:17:58.066453+0000","last_clean":"2024-07-27T15:17:58.066453+0000","last_became_active":"2024-07-27T14:42:32.853532+0000","last_became_peered":"2024-07-27T14:42:32.853532+0000","last_unstale":"2024-07-27T15:17:58.066453+0000","last_undegraded":"2024-07-27T15:17:58.066453+0000","last_fullsized":"2024-07-27T15:17:58.066453+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:58.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:58.064870+0000","last_clean_scrub_stamp":"2024-07-27T15:17:58.064870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:31:26.630429+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411772,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76920,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294920,"statfs":{"total":96636764160,"available":96557998080,"internally_reserved":0,"allocated":5611520,"data_stored":2659637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":5611520,"data_stored":2659623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509878,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":5484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":5615616,"data_stored":2659642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509882,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73140,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298700,"statfs":{"total":96636764160,"available":96561868800,"internally_reserved":0,"allocated":6008832,"data_stored":3057458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509884,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72748,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299092,"statfs":{"total":96636764160,"available":96562270208,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76912,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294928,"statfs":{"total":96636764160,"available":96558006272,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575284,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77304,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294536,"statfs":{"total":96636764160,"available":96557604864,"internally_reserved":0,"allocated":6008832,"data_stored":3057458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575287,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T15:18:01.459 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.459 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.459 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.459 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.459 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.460 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.461 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-07-27T14:41:16.982882+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=41, tm_sec=16, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.462 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.463 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.463 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.463 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.463 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-07-27T14:42:27.068054+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=5, tm_yday=209, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=27, tm_hour=15, tm_min=17, tm_sec=53, tm_wday=5, tm_yday=209, tm_isdst=0) 2024-07-27T15:18:01.463 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-07-27T15:18:05.012 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:07.878 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.015 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:07.881 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.031 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:07.897 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.062 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:07.913 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.063 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:07.914 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.080 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:07.932 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.084 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:07.949 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.087 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:07.953 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:05.106 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:07.972 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.020 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:12.885 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.024 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:12.890 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.035 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:12.901 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.073 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:12.939 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.073 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:12.925 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.075 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:12.927 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.078 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:12.944 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.085 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:12.937 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:10.089 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:12.955 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:21.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T15:18:21.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.566+0000 7f3a86cde640 10 monclient: get_monmap_and_config 2024-07-27T15:18:21.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.566+0000 7f3a86cde640 10 monclient: build_initial_monmap 2024-07-27T15:18:21.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.566+0000 7f3a86cde640 10 monclient: monmap: 2024-07-27T15:18:21.567 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:21.567 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:18:21.567288+0000 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:18:21.567288+0000 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:18:21.568 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.566+0000 7f3a86cde640 1 Processor -- start 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 -- start start 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient: init 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3a800b3630 0x7f3a80143230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-c con 0x7f3a800b3630 addr v2:172.21.15.112:3300/0 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a80143770 addr v1:172.21.15.87:6789/0 2024-07-27T15:18:21.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a8014d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-b con 0x7f3a800aab60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a800aa8a0 con 0x7f3a80143770 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 -- 0x7f3a800b3a10 con 0x7f3a800b3630 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f3a80142da0 con 0x7f3a800aab60 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.567+0000 7f3a86cde640 10 monclient(hunting): authenticate will time out at 3441.593679s 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a8527b640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a8014d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a7ffff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3a80143770 0x7f3a8013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51636/0 (socket says 172.21.15.87:51636) 2024-07-27T15:18:21.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a7ffff640 1 -- 172.21.15.87:0/75703881 learned_addr learned my addr 172.21.15.87:0/75703881 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a84a7a640 1 --2- 172.21.15.87:0/75703881 >> v2:172.21.15.112:3300/0 conn(0x7f3a800b3630 0x7f3a80143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a8527b640 10 monclient(hunting): get_auth_request con 0x7f3a800aab60 auth_method 0 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a8527b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a8527b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a8527b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a84a7a640 10 monclient(hunting): get_auth_request con 0x7f3a800b3630 auth_method 0 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a84a7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a84a7a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.568+0000 7f3a84a7a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:21.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:21.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:21.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:21.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:21.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.569+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 1 -- 172.21.15.87:0/75703881 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172614234 0 0) 0x7f3a800aa8a0 con 0x7f3a80143770 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 10 monclient(hunting): _negotiate 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 10 monclient(hunting): my global_id is 14838 2024-07-27T15:18:21.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a7f7fe640 1 -- 172.21.15.87:0/75703881 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a70003520 con 0x7f3a80143770 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 10 monclient(hunting): handle_auth_done global_id 15584 payload 274 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 1 -- 172.21.15.87:0/75703881 >> v2:172.21.15.112:3300/0 conn(0x7f3a800b3630 msgr2=0x7f3a80143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 1 --2- 172.21.15.87:0/75703881 >> v2:172.21.15.112:3300/0 conn(0x7f3a800b3630 0x7f3a80143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 1 -- 172.21.15.87:0/75703881 >> v1:172.21.15.87:6789/0 conn(0x7f3a80143770 legacy=0x7f3a8013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 10 monclient: _finish_hunting 0 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 1 monclient: found mon.noname-b 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.570+0000 7f3a8527b640 1 -- 172.21.15.87:0/75703881 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a80140620 con 0x7f3a800aab60 2024-07-27T15:18:21.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.572+0000 7f3a8527b640 1 --2- 172.21.15.87:0/75703881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a8014d410 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009d90 tx=0x7f3a7802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e79a5ca2f25a92 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 1 -- 172.21.15.87:0/75703881 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a78039070 con 0x7f3a800aab60 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: dump: 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.578 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: _finish_auth 0 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: _check_auth_tickets 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 1 -- 172.21.15.87:0/75703881 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7802cbc0 con 0x7f3a800aab60 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 1 -- 172.21.15.87:0/75703881 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a78034440 con 0x7f3a800aab60 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 5 monclient: authenticate success, global_id 15584 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 10 monclient: get_monmap_and_config success 2024-07-27T15:18:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 4 set_mon_vals no callback set 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a7f7fe640 10 monclient: dump: 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 10 monclient: shutdown 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 1 -- 172.21.15.87:0/75703881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 msgr2=0x7f3a8014d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 1 --2- 172.21.15.87:0/75703881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a8014d410 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009d90 tx=0x7f3a7802be10 comp rx=0 tx=0).stop 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 1 -- 172.21.15.87:0/75703881 shutdown_connections 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 1 --2- 172.21.15.87:0/75703881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a8014d410 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.576+0000 7f3a86cde640 1 --2- 172.21.15.87:0/75703881 >> v2:172.21.15.112:3300/0 conn(0x7f3a800b3630 0x7f3a80143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 1 -- 172.21.15.87:0/75703881 >> 172.21.15.87:0/75703881 conn(0x7f3a800730a0 msgr2=0x7f3a80067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 1 -- 172.21.15.87:0/75703881 shutdown_connections 2024-07-27T15:18:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 1 -- 172.21.15.87:0/75703881 wait complete. 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 10 monclient: build_initial_monmap 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 10 monclient: monmap: 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:18:21.586 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 1 librados: starting msgr at 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.577+0000 7f3a86cde640 1 librados: starting objecter 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.578+0000 7f3a86cde640 1 Processor -- start 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.578+0000 7f3a86cde640 1 -- start start 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.578+0000 7f3a86cde640 1 librados: setting wanted keys 2024-07-27T15:18:21.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.578+0000 7f3a86cde640 1 librados: calling monclient init 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.578+0000 7f3a86cde640 10 monclient: init 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-c con 0x7f3a800aab60 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a800b3630 0x7f3a801d8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-a con 0x7f3a800b3630 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a80143770 0x7f3a801d8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): picked mon.noname-b con 0x7f3a80143770 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3a800b4840 con 0x7f3a800b3630 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:21.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3a800b46c0 con 0x7f3a80143770 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7f3a800b49c0 con 0x7f3a800aab60 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a86cde640 10 monclient(hunting): authenticate will time out at 3441.605129s 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a84a7a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a7ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a800b3630 0x7f3a801d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a8527b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a80143770 0x7f3a801d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.579+0000 7f3a84a7a640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34714/0 (socket says 172.21.15.87:34714) 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 1 -- 172.21.15.87:0/3519326559 learned_addr learned my addr 172.21.15.87:0/3519326559 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): get_auth_request con 0x7f3a800aab60 auth_method 0 2024-07-27T15:18:21.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): get_auth_request con 0x7f3a800b3630 auth_method 0 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): get_auth_request con 0x7f3a80143770 auth_method 0 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.580+0000 7f3a8527b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:21.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 10 monclient(hunting): handle_auth_done global_id 15587 payload 824 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 -- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a80143770 msgr2=0x7f3a801d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a80143770 0x7f3a801d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 -- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a800b3630 msgr2=0x7f3a801d8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a800b3630 0x7f3a801d8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 10 monclient: _finish_hunting 0 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 monclient: found mon.noname-c 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.583+0000 7f3a84a7a640 1 -- 172.21.15.87:0/3519326559 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a800aeec0 con 0x7f3a800aab60 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.585+0000 7f3a84a7a640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400e9e0 tx=0x7f3a7400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.585+0000 7f3a7d7fa640 10 client.?.objecter ms_handle_connect 0x7f3a800aab60 2024-07-27T15:18:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.585+0000 7f3a7d7fa640 10 client.?.objecter resend_mon_ops 2024-07-27T15:18:21.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74017070 con 0x7f3a800aab60 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: dump: 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:21.595 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: _finish_auth 0 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: _check_auth_tickets 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7400c9d0 con 0x7f3a800aab60 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 5 monclient: authenticate success, global_id 15587 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 10 monclient: _renew_subs 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a800af1a0 con 0x7f3a800aab60 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400cdc0 con 0x7f3a800aab60 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:21.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7d7fa640 10 monclient: dump: 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a7effd640 4 set_mon_vals no callback set 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 10 client.15587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 10 monclient: _renew_subs 2024-07-27T15:18:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a800ae1a0 con 0x7f3a800aab60 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.588+0000 7f3a86cde640 1 librados: init done 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.589+0000 7f3a86cde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.589+0000 7f3a86cde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.589+0000 7f3a86cde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.589+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80143230 con 0x7f3a800aab60 2024-07-27T15:18:21.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.837+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3a7401a070 con 0x7f3a800aab60 2024-07-27T15:18:21.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.837+0000 7f3a7d7fa640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 0x7f3a6406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:21.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.837+0000 7f3a7ffff640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 0x7f3a6406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:21.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.837+0000 7f3a7ffff640 10 monclient: get_auth_request con 0x7f3a6406d7c0 auth_method 0 2024-07-27T15:18:21.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.838+0000 7f3a7d7fa640 10 client.15587.objecter ms_handle_connect 0x7f3a6406d7c0 2024-07-27T15:18:21.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.838+0000 7f3a7ffff640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 0x7f3a6406fc80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3a80140550 tx=0x7f3a6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7f3a74057a00 con 0x7f3a800aab60 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 10 client.15587.objecter ms_dispatch 0x7f3a80000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 3 client.15587.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 3 client.15587.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 20 client.15587.objecter dump_active .. 0 homeless 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7f3a74059960 con 0x7f3a800aab60 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:21.850+0000 7f3a7d7fa640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:18:22.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.024+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 --> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3a80141dd0 con 0x7f3a6406d7c0 2024-07-27T15:18:22.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.027+0000 7f3a7d7fa640 1 -- 172.21.15.87:0/3519326559 <== mgr.4104 v2:172.21.15.112:6800/1547416426 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116299 (secure 0 0 0) 0x7f3a80141dd0 con 0x7f3a6406d7c0 2024-07-27T15:18:22.030 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-07-27T15:18:22.031 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-07-27T15:18:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 10 librados: watch_flush enter 2024-07-27T15:18:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 10 librados: watch_flush exit 2024-07-27T15:18:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 20 client.15587.objecter shutdown clearing up homeless session... 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 10 client.15587.objecter successfully canceled tick 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 msgr2=0x7f3a6406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 0x7f3a6406fc80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3a80140550 tx=0x7f3a6c009040 comp rx=0 tx=0).stop 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 10 monclient: shutdown 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 msgr2=0x7f3a800ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.029+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400e9e0 tx=0x7f3a7400eea0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 shutdown_connections 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7f3a6406d7c0 0x7f3a6406fc80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a80143770 0x7f3a801d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a800b3630 0x7f3a801d8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 --2- 172.21.15.87:0/3519326559 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a800aab60 0x7f3a800ad090 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 >> 172.21.15.87:0/3519326559 conn(0x7f3a800730a0 msgr2=0x7f3a80140a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 shutdown_connections 2024-07-27T15:18:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 -- 172.21.15.87:0/3519326559 wait complete. 2024-07-27T15:18:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.030+0000 7f3a86cde640 1 librados: shutdown 2024-07-27T15:18:22.042 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":2592,"stamp":"2024-07-27T15:18:20.499783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":122,"num_write_kb":1381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":595264,"kb_used_data":44608,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754379456,"statfs":{"total":773094113280,"available":772484562944,"internally_reserved":0,"allocated":45678592,"data_stored":22072672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563793456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002973"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":2827,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.591566+0000","last_change":"2024-07-27T15:18:02.592369+0000","last_active":"2024-07-27T15:18:02.591566+0000","last_peered":"2024-07-27T15:18:02.591566+0000","last_clean":"2024-07-27T15:18:02.591566+0000","last_became_active":"2024-07-27T14:42:41.574286+0000","last_became_peered":"2024-07-27T14:42:41.574286+0000","last_unstale":"2024-07-27T15:18:02.591566+0000","last_undegraded":"2024-07-27T15:18:02.591566+0000","last_fullsized":"2024-07-27T15:18:02.591566+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.590064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.590064+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.590064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:51:18.381951+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":2835,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.579606+0000","last_change":"2024-07-27T15:18:02.580357+0000","last_active":"2024-07-27T15:18:02.579606+0000","last_peered":"2024-07-27T15:18:02.579606+0000","last_clean":"2024-07-27T15:18:02.579606+0000","last_became_active":"2024-07-27T14:42:33.130123+0000","last_became_peered":"2024-07-27T14:42:33.130123+0000","last_unstale":"2024-07-27T15:18:02.579606+0000","last_undegraded":"2024-07-27T15:18:02.579606+0000","last_fullsized":"2024-07-27T15:18:02.579606+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.578115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.578115+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.578115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:40:29.337394+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2841,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.427994+0000","last_change":"2024-07-27T15:18:00.428816+0000","last_active":"2024-07-27T15:18:00.427994+0000","last_peered":"2024-07-27T15:18:00.427994+0000","last_clean":"2024-07-27T15:18:00.427994+0000","last_became_active":"2024-07-27T14:42:32.196113+0000","last_became_peered":"2024-07-27T14:42:32.196113+0000","last_unstale":"2024-07-27T15:18:00.427994+0000","last_undegraded":"2024-07-27T15:18:00.427994+0000","last_fullsized":"2024-07-27T15:18:00.427994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:00.426386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:00.426386+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.426386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:18:56.165403+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2840,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.373375+0000","last_change":"2024-07-27T15:18:02.374174+0000","last_active":"2024-07-27T15:18:02.373375+0000","last_peered":"2024-07-27T15:18:02.373375+0000","last_clean":"2024-07-27T15:18:02.373375+0000","last_became_active":"2024-07-27T14:42:33.550008+0000","last_became_peered":"2024-07-27T14:42:33.550008+0000","last_unstale":"2024-07-27T15:18:02.373375+0000","last_undegraded":"2024-07-27T15:18:02.373375+0000","last_fullsized":"2024-07-27T15:18:02.373375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.371742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.371742+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.371742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:39:35.472108+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":2840,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.618037+0000","last_change":"2024-07-27T15:18:03.618813+0000","last_active":"2024-07-27T15:18:03.618037+0000","last_peered":"2024-07-27T15:18:03.618037+0000","last_clean":"2024-07-27T15:18:03.618037+0000","last_became_active":"2024-07-27T14:42:33.554937+0000","last_became_peered":"2024-07-27T14:42:33.554937+0000","last_unstale":"2024-07-27T15:18:03.618037+0000","last_undegraded":"2024-07-27T15:18:03.618037+0000","last_fullsized":"2024-07-27T15:18:03.618037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.616493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.616493+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.616493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:32:43.536457+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2813,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.341058+0000","last_change":"2024-07-27T15:17:57.341860+0000","last_active":"2024-07-27T15:17:57.341058+0000","last_peered":"2024-07-27T15:17:57.341058+0000","last_clean":"2024-07-27T15:17:57.341058+0000","last_became_active":"2024-07-27T14:42:43.877378+0000","last_became_peered":"2024-07-27T14:42:43.877378+0000","last_unstale":"2024-07-27T15:17:57.341058+0000","last_undegraded":"2024-07-27T15:17:57.341058+0000","last_fullsized":"2024-07-27T15:17:57.341058+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.339321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.339321+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.339321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:37:21.811853+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":2826,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.056852+0000","last_change":"2024-07-27T15:18:00.057616+0000","last_active":"2024-07-27T15:18:00.056852+0000","last_peered":"2024-07-27T15:18:00.056852+0000","last_clean":"2024-07-27T15:18:00.056852+0000","last_became_active":"2024-07-27T14:42:35.220195+0000","last_became_peered":"2024-07-27T14:42:35.220195+0000","last_unstale":"2024-07-27T15:18:00.056852+0000","last_undegraded":"2024-07-27T15:18:00.056852+0000","last_fullsized":"2024-07-27T15:18:00.056852+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:00.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:00.052908+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:42:32.120583+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2843,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.381296+0000","last_change":"2024-07-27T15:18:03.382074+0000","last_active":"2024-07-27T15:18:03.381296+0000","last_peered":"2024-07-27T15:18:03.381296+0000","last_clean":"2024-07-27T15:18:03.381296+0000","last_became_active":"2024-07-27T14:42:33.867602+0000","last_became_peered":"2024-07-27T14:42:33.867602+0000","last_unstale":"2024-07-27T15:18:03.381296+0000","last_undegraded":"2024-07-27T15:18:03.381296+0000","last_fullsized":"2024-07-27T15:18:03.381296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.379391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.379391+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.379391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:49:23.839996+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":2836,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:55.588628+0000","last_change":"2024-07-27T15:17:55.589452+0000","last_active":"2024-07-27T15:17:55.588628+0000","last_peered":"2024-07-27T15:17:55.588628+0000","last_clean":"2024-07-27T15:17:55.588628+0000","last_became_active":"2024-07-27T14:42:34.746905+0000","last_became_peered":"2024-07-27T14:42:34.746905+0000","last_unstale":"2024-07-27T15:17:55.588628+0000","last_undegraded":"2024-07-27T15:17:55.588628+0000","last_fullsized":"2024-07-27T15:17:55.588628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:55.587059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:55.587059+0000","last_clean_scrub_stamp":"2024-07-27T15:17:55.587059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:03.467232+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":2837,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:04.604248+0000","last_change":"2024-07-27T15:18:04.604995+0000","last_active":"2024-07-27T15:18:04.604248+0000","last_peered":"2024-07-27T15:18:04.604248+0000","last_clean":"2024-07-27T15:18:04.604248+0000","last_became_active":"2024-07-27T14:42:37.287685+0000","last_became_peered":"2024-07-27T14:42:37.287685+0000","last_unstale":"2024-07-27T15:18:04.604248+0000","last_undegraded":"2024-07-27T15:18:04.604248+0000","last_fullsized":"2024-07-27T15:18:04.604248+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:04.602577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:04.602577+0000","last_clean_scrub_stamp":"2024-07-27T15:18:04.602577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:02:03.135027+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":2823,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.057712+0000","last_change":"2024-07-27T15:18:02.058490+0000","last_active":"2024-07-27T15:18:02.057712+0000","last_peered":"2024-07-27T15:18:02.057712+0000","last_clean":"2024-07-27T15:18:02.057712+0000","last_became_active":"2024-07-27T14:42:35.354552+0000","last_became_peered":"2024-07-27T14:42:35.354552+0000","last_unstale":"2024-07-27T15:18:02.057712+0000","last_undegraded":"2024-07-27T15:18:02.057712+0000","last_fullsized":"2024-07-27T15:18:02.057712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.056091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.056091+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.056091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T03:01:28.781756+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.928748+0000","last_change":"2024-07-27T15:18:03.929499+0000","last_active":"2024-07-27T15:18:03.928748+0000","last_peered":"2024-07-27T15:18:03.928748+0000","last_clean":"2024-07-27T15:18:03.928748+0000","last_became_active":"2024-07-27T14:42:34.568719+0000","last_became_peered":"2024-07-27T14:42:34.568719+0000","last_unstale":"2024-07-27T15:18:03.928748+0000","last_undegraded":"2024-07-27T15:18:03.928748+0000","last_fullsized":"2024-07-27T15:18:03.928748+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.927208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.927208+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.927208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:26:28.023291+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1468'78","reported_seq":2942,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:56.423173+0000","last_change":"2024-07-27T15:17:56.424691+0000","last_active":"2024-07-27T15:17:56.423173+0000","last_peered":"2024-07-27T15:17:56.423173+0000","last_clean":"2024-07-27T15:17:56.423173+0000","last_became_active":"2024-07-27T14:41:03.582590+0000","last_became_peered":"2024-07-27T14:41:03.582590+0000","last_unstale":"2024-07-27T15:17:56.423173+0000","last_undegraded":"2024-07-27T15:17:56.423173+0000","last_fullsized":"2024-07-27T15:17:56.423173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'78","last_scrub_stamp":"2024-07-27T15:17:56.420345+0000","last_deep_scrub":"1468'78","last_deep_scrub_stamp":"2024-07-27T15:17:56.420345+0000","last_clean_scrub_stamp":"2024-07-27T15:17:56.420345+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:46:32.650722+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"1468'2","reported_seq":2861,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.598660+0000","last_change":"2024-07-27T15:18:00.599544+0000","last_active":"2024-07-27T15:18:00.598660+0000","last_peered":"2024-07-27T15:18:00.598660+0000","last_clean":"2024-07-27T15:18:00.598660+0000","last_became_active":"2024-07-27T14:41:18.896458+0000","last_became_peered":"2024-07-27T14:41:18.896458+0000","last_unstale":"2024-07-27T15:18:00.598660+0000","last_undegraded":"2024-07-27T15:18:00.598660+0000","last_fullsized":"2024-07-27T15:18:00.598660+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'2","last_scrub_stamp":"2024-07-27T15:18:00.597028+0000","last_deep_scrub":"1468'2","last_deep_scrub_stamp":"2024-07-27T15:18:00.597028+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.597028+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:22:30.547913+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0.068506337,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":2839,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:05.608255+0000","last_change":"2024-07-27T15:18:05.608255+0000","last_active":"2024-07-27T15:18:05.608255+0000","last_peered":"2024-07-27T15:18:05.608255+0000","last_clean":"2024-07-27T15:18:05.608255+0000","last_became_active":"2024-07-27T14:42:37.189428+0000","last_became_peered":"2024-07-27T14:42:37.189428+0000","last_unstale":"2024-07-27T15:18:05.608255+0000","last_undegraded":"2024-07-27T15:18:05.608255+0000","last_fullsized":"2024-07-27T15:18:05.608255+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:05.604519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:05.604519+0000","last_clean_scrub_stamp":"2024-07-27T15:18:05.604519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:12:28.013433+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2859,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.583693+0000","last_change":"2024-07-27T15:17:57.584731+0000","last_active":"2024-07-27T15:17:57.583693+0000","last_peered":"2024-07-27T15:17:57.583693+0000","last_clean":"2024-07-27T15:17:57.583693+0000","last_became_active":"2024-07-27T14:41:18.490480+0000","last_became_peered":"2024-07-27T14:41:18.490480+0000","last_unstale":"2024-07-27T15:17:57.583693+0000","last_undegraded":"2024-07-27T15:17:57.583693+0000","last_fullsized":"2024-07-27T15:17:57.583693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.582075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.582075+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.582075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T19:18:38.448714+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.112541057,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2843,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.425798+0000","last_change":"2024-07-27T15:18:01.426660+0000","last_active":"2024-07-27T15:18:01.425798+0000","last_peered":"2024-07-27T15:18:01.425798+0000","last_clean":"2024-07-27T15:18:01.425798+0000","last_became_active":"2024-07-27T14:42:32.825779+0000","last_became_peered":"2024-07-27T14:42:32.825779+0000","last_unstale":"2024-07-27T15:18:01.425798+0000","last_undegraded":"2024-07-27T15:18:01.425798+0000","last_fullsized":"2024-07-27T15:18:01.425798+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.424119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.424119+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.424119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:21:59.243674+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2836,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.907033+0000","last_change":"2024-07-27T15:18:00.907794+0000","last_active":"2024-07-27T15:18:00.907033+0000","last_peered":"2024-07-27T15:18:00.907033+0000","last_clean":"2024-07-27T15:18:00.907033+0000","last_became_active":"2024-07-27T14:41:18.386663+0000","last_became_peered":"2024-07-27T14:41:18.386663+0000","last_unstale":"2024-07-27T15:18:00.907033+0000","last_undegraded":"2024-07-27T15:18:00.907033+0000","last_fullsized":"2024-07-27T15:18:00.907033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:00.905434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:00.905434+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.905434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:28:09.860608+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0.061484742000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2826,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:04.593280+0000","last_change":"2024-07-27T15:18:04.594028+0000","last_active":"2024-07-27T15:18:04.593280+0000","last_peered":"2024-07-27T15:18:04.593280+0000","last_clean":"2024-07-27T15:18:04.593280+0000","last_became_active":"2024-07-27T14:43:03.360033+0000","last_became_peered":"2024-07-27T14:43:03.360033+0000","last_unstale":"2024-07-27T15:18:04.593280+0000","last_undegraded":"2024-07-27T15:18:04.593280+0000","last_fullsized":"2024-07-27T15:18:04.593280+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:04.591788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:04.591788+0000","last_clean_scrub_stamp":"2024-07-27T15:18:04.591788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:41:53.656080+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"1468'3","reported_seq":2862,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:59.563826+0000","last_change":"2024-07-27T15:17:59.565037+0000","last_active":"2024-07-27T15:17:59.563826+0000","last_peered":"2024-07-27T15:17:59.563826+0000","last_clean":"2024-07-27T15:17:59.563826+0000","last_became_active":"2024-07-27T14:41:18.722358+0000","last_became_peered":"2024-07-27T14:41:18.722358+0000","last_unstale":"2024-07-27T15:17:59.563826+0000","last_undegraded":"2024-07-27T15:17:59.563826+0000","last_fullsized":"2024-07-27T15:17:59.563826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'3","last_scrub_stamp":"2024-07-27T15:17:59.562229+0000","last_deep_scrub":"1468'3","last_deep_scrub_stamp":"2024-07-27T15:17:59.562229+0000","last_clean_scrub_stamp":"2024-07-27T15:17:59.562229+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T03:14:19.373361+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0.10903597,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2829,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.536055+0000","last_change":"2024-07-27T15:18:00.536055+0000","last_active":"2024-07-27T15:18:00.536055+0000","last_peered":"2024-07-27T15:18:00.536055+0000","last_clean":"2024-07-27T15:18:00.536055+0000","last_became_active":"2024-07-27T14:42:33.415149+0000","last_became_peered":"2024-07-27T14:42:33.415149+0000","last_unstale":"2024-07-27T15:18:00.536055+0000","last_undegraded":"2024-07-27T15:18:00.536055+0000","last_fullsized":"2024-07-27T15:18:00.536055+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:00.533689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:00.533689+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.533689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:51:53.875998+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2863,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:59.468755+0000","last_change":"2024-07-27T15:17:59.469547+0000","last_active":"2024-07-27T15:17:59.468755+0000","last_peered":"2024-07-27T15:17:59.468755+0000","last_clean":"2024-07-27T15:17:59.468755+0000","last_became_active":"2024-07-27T14:41:18.731804+0000","last_became_peered":"2024-07-27T14:41:18.731804+0000","last_unstale":"2024-07-27T15:17:59.468755+0000","last_undegraded":"2024-07-27T15:17:59.468755+0000","last_fullsized":"2024-07-27T15:17:59.468755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:59.467019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:59.467019+0000","last_clean_scrub_stamp":"2024-07-27T15:17:59.467019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:59:58.474706+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.062539163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2829,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.610506+0000","last_change":"2024-07-27T15:18:03.611262+0000","last_active":"2024-07-27T15:18:03.610506+0000","last_peered":"2024-07-27T15:18:03.610506+0000","last_clean":"2024-07-27T15:18:03.610506+0000","last_became_active":"2024-07-27T14:42:57.112921+0000","last_became_peered":"2024-07-27T14:42:57.112921+0000","last_unstale":"2024-07-27T15:18:03.610506+0000","last_undegraded":"2024-07-27T15:18:03.610506+0000","last_fullsized":"2024-07-27T15:18:03.610506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.609048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.609048+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.609048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:13:41.262992+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2856,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:58.549169+0000","last_change":"2024-07-27T15:17:58.549939+0000","last_active":"2024-07-27T15:17:58.549169+0000","last_peered":"2024-07-27T15:17:58.549169+0000","last_clean":"2024-07-27T15:17:58.549169+0000","last_became_active":"2024-07-27T14:41:18.690235+0000","last_became_peered":"2024-07-27T14:41:18.690235+0000","last_unstale":"2024-07-27T15:17:58.549169+0000","last_undegraded":"2024-07-27T15:17:58.549169+0000","last_fullsized":"2024-07-27T15:17:58.549169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:58.547646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:58.547646+0000","last_clean_scrub_stamp":"2024-07-27T15:17:58.547646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:50:09.779975+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.071187715999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2829,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.540870+0000","last_change":"2024-07-27T15:18:01.540870+0000","last_active":"2024-07-27T15:18:01.540870+0000","last_peered":"2024-07-27T15:18:01.540870+0000","last_clean":"2024-07-27T15:18:01.540870+0000","last_became_active":"2024-07-27T14:42:36.941428+0000","last_became_peered":"2024-07-27T14:42:36.941428+0000","last_unstale":"2024-07-27T15:18:01.540870+0000","last_undegraded":"2024-07-27T15:18:01.540870+0000","last_fullsized":"2024-07-27T15:18:01.540870+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.538632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.538632+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.538632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:30:45.686152+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2851,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:58.417263+0000","last_change":"2024-07-27T15:17:58.417263+0000","last_active":"2024-07-27T15:17:58.417263+0000","last_peered":"2024-07-27T15:17:58.417263+0000","last_clean":"2024-07-27T15:17:58.417263+0000","last_became_active":"2024-07-27T14:41:18.927187+0000","last_became_peered":"2024-07-27T14:41:18.927187+0000","last_unstale":"2024-07-27T15:17:58.417263+0000","last_undegraded":"2024-07-27T15:17:58.417263+0000","last_fullsized":"2024-07-27T15:17:58.417263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:58.413710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:58.413710+0000","last_clean_scrub_stamp":"2024-07-27T15:17:58.413710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T00:53:53.852327+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.065812075999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2834,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:00.394051+0000","last_change":"2024-07-27T15:18:00.394848+0000","last_active":"2024-07-27T15:18:00.394051+0000","last_peered":"2024-07-27T15:18:00.394051+0000","last_clean":"2024-07-27T15:18:00.394051+0000","last_became_active":"2024-07-27T14:42:32.329977+0000","last_became_peered":"2024-07-27T14:42:32.329977+0000","last_unstale":"2024-07-27T15:18:00.394051+0000","last_undegraded":"2024-07-27T15:18:00.394051+0000","last_fullsized":"2024-07-27T15:18:00.394051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:00.392356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:00.392356+0000","last_clean_scrub_stamp":"2024-07-27T15:18:00.392356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:07:23.498637+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2854,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.557711+0000","last_change":"2024-07-27T15:18:01.557711+0000","last_active":"2024-07-27T15:18:01.557711+0000","last_peered":"2024-07-27T15:18:01.557711+0000","last_clean":"2024-07-27T15:18:01.557711+0000","last_became_active":"2024-07-27T14:41:19.202611+0000","last_became_peered":"2024-07-27T14:41:19.202611+0000","last_unstale":"2024-07-27T15:18:01.557711+0000","last_undegraded":"2024-07-27T15:18:01.557711+0000","last_fullsized":"2024-07-27T15:18:01.557711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.555500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.555500+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.555500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:48:57.360003+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.073274100999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1468'8","reported_seq":2842,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:59.061049+0000","last_change":"2024-07-27T15:17:59.062048+0000","last_active":"2024-07-27T15:17:59.061049+0000","last_peered":"2024-07-27T15:17:59.061049+0000","last_clean":"2024-07-27T15:17:59.061049+0000","last_became_active":"2024-07-27T14:42:34.791352+0000","last_became_peered":"2024-07-27T14:42:34.791352+0000","last_unstale":"2024-07-27T15:17:59.061049+0000","last_undegraded":"2024-07-27T15:17:59.061049+0000","last_fullsized":"2024-07-27T15:17:59.061049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"1468'8","last_scrub_stamp":"2024-07-27T15:17:59.059180+0000","last_deep_scrub":"1468'8","last_deep_scrub_stamp":"2024-07-27T15:17:59.059180+0000","last_clean_scrub_stamp":"2024-07-27T15:17:59.059180+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:58:03.698313+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":2825,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.075202+0000","last_change":"2024-07-27T15:18:01.075949+0000","last_active":"2024-07-27T15:18:01.075202+0000","last_peered":"2024-07-27T15:18:01.075202+0000","last_clean":"2024-07-27T15:18:01.075202+0000","last_became_active":"2024-07-27T14:42:35.248761+0000","last_became_peered":"2024-07-27T14:42:35.248761+0000","last_unstale":"2024-07-27T15:18:01.075202+0000","last_undegraded":"2024-07-27T15:18:01.075202+0000","last_fullsized":"2024-07-27T15:18:01.075202+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.073635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.073635+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.073635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:53.805317+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.459976+0000","last_change":"2024-07-27T15:18:03.460757+0000","last_active":"2024-07-27T15:18:03.459976+0000","last_peered":"2024-07-27T15:18:03.459976+0000","last_clean":"2024-07-27T15:18:03.459976+0000","last_became_active":"2024-07-27T14:42:53.131434+0000","last_became_peered":"2024-07-27T14:42:53.131434+0000","last_unstale":"2024-07-27T15:18:03.459976+0000","last_undegraded":"2024-07-27T15:18:03.459976+0000","last_fullsized":"2024-07-27T15:18:03.459976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.458367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.458367+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.458367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:37:43.918670+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2832,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.419708+0000","last_change":"2024-07-27T15:18:01.420514+0000","last_active":"2024-07-27T15:18:01.419708+0000","last_peered":"2024-07-27T15:18:01.419708+0000","last_clean":"2024-07-27T15:18:01.419708+0000","last_became_active":"2024-07-27T14:42:32.755122+0000","last_became_peered":"2024-07-27T14:42:32.755122+0000","last_unstale":"2024-07-27T15:18:01.419708+0000","last_undegraded":"2024-07-27T15:18:01.419708+0000","last_fullsized":"2024-07-27T15:18:01.419708+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.418051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.418051+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.418051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:56:13.392447+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2816,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:04.935746+0000","last_change":"2024-07-27T15:18:04.936533+0000","last_active":"2024-07-27T15:18:04.935746+0000","last_peered":"2024-07-27T15:18:04.935746+0000","last_clean":"2024-07-27T15:18:04.935746+0000","last_became_active":"2024-07-27T14:42:34.647534+0000","last_became_peered":"2024-07-27T14:42:34.647534+0000","last_unstale":"2024-07-27T15:18:04.935746+0000","last_undegraded":"2024-07-27T15:18:04.935746+0000","last_fullsized":"2024-07-27T15:18:04.935746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:04.934208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:04.934208+0000","last_clean_scrub_stamp":"2024-07-27T15:18:04.934208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T20:54:58.645941+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":2838,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:06.588610+0000","last_change":"2024-07-27T15:18:06.589358+0000","last_active":"2024-07-27T15:18:06.588610+0000","last_peered":"2024-07-27T15:18:06.588610+0000","last_clean":"2024-07-27T15:18:06.588610+0000","last_became_active":"2024-07-27T14:42:37.973009+0000","last_became_peered":"2024-07-27T14:42:37.973009+0000","last_unstale":"2024-07-27T15:18:06.588610+0000","last_undegraded":"2024-07-27T15:18:06.588610+0000","last_fullsized":"2024-07-27T15:18:06.588610+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:06.587154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:06.587154+0000","last_clean_scrub_stamp":"2024-07-27T15:18:06.587154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T21:47:19.764579+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2833,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:59.397372+0000","last_change":"2024-07-27T15:17:59.398130+0000","last_active":"2024-07-27T15:17:59.397372+0000","last_peered":"2024-07-27T15:17:59.397372+0000","last_clean":"2024-07-27T15:17:59.397372+0000","last_became_active":"2024-07-27T14:42:32.194637+0000","last_became_peered":"2024-07-27T14:42:32.194637+0000","last_unstale":"2024-07-27T15:17:59.397372+0000","last_undegraded":"2024-07-27T15:17:59.397372+0000","last_fullsized":"2024-07-27T15:17:59.397372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:59.395776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:59.395776+0000","last_clean_scrub_stamp":"2024-07-27T15:17:59.395776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:47:00.291062+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:01.947007+0000","last_change":"2024-07-27T15:18:01.947755+0000","last_active":"2024-07-27T15:18:01.947007+0000","last_peered":"2024-07-27T15:18:01.947007+0000","last_clean":"2024-07-27T15:18:01.947007+0000","last_became_active":"2024-07-27T14:42:30.517011+0000","last_became_peered":"2024-07-27T14:42:30.517011+0000","last_unstale":"2024-07-27T15:18:01.947007+0000","last_undegraded":"2024-07-27T15:18:01.947007+0000","last_fullsized":"2024-07-27T15:18:01.947007+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:01.945461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:01.945461+0000","last_clean_scrub_stamp":"2024-07-27T15:18:01.945461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:33:13.556920+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2830,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.423874+0000","last_change":"2024-07-27T15:18:02.424869+0000","last_active":"2024-07-27T15:18:02.423874+0000","last_peered":"2024-07-27T15:18:02.423874+0000","last_clean":"2024-07-27T15:18:02.423874+0000","last_became_active":"2024-07-27T14:42:44.426410+0000","last_became_peered":"2024-07-27T14:42:44.426410+0000","last_unstale":"2024-07-27T15:18:02.423874+0000","last_undegraded":"2024-07-27T15:18:02.423874+0000","last_fullsized":"2024-07-27T15:18:02.423874+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.422291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.422291+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.422291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:30:39.030048+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:03.075298+0000","last_change":"2024-07-27T15:18:03.076073+0000","last_active":"2024-07-27T15:18:03.075298+0000","last_peered":"2024-07-27T15:18:03.075298+0000","last_clean":"2024-07-27T15:18:03.075298+0000","last_became_active":"2024-07-27T14:43:09.707672+0000","last_became_peered":"2024-07-27T14:43:09.707672+0000","last_unstale":"2024-07-27T15:18:03.075298+0000","last_undegraded":"2024-07-27T15:18:03.075298+0000","last_fullsized":"2024-07-27T15:18:03.075298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:03.073732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:03.073732+0000","last_clean_scrub_stamp":"2024-07-27T15:18:03.073732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:05:34.455220+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:57.080606+0000","last_change":"2024-07-27T15:17:57.081377+0000","last_active":"2024-07-27T15:17:57.080606+0000","last_peered":"2024-07-27T15:17:57.080606+0000","last_clean":"2024-07-27T15:17:57.080606+0000","last_became_active":"2024-07-27T14:42:32.384600+0000","last_became_peered":"2024-07-27T14:42:32.384600+0000","last_unstale":"2024-07-27T15:17:57.080606+0000","last_undegraded":"2024-07-27T15:17:57.080606+0000","last_fullsized":"2024-07-27T15:17:57.080606+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:57.079054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:57.079054+0000","last_clean_scrub_stamp":"2024-07-27T15:17:57.079054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T01:37:13.227424+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2815,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:18:02.953153+0000","last_change":"2024-07-27T15:18:02.953900+0000","last_active":"2024-07-27T15:18:02.953153+0000","last_peered":"2024-07-27T15:18:02.953153+0000","last_clean":"2024-07-27T15:18:02.953153+0000","last_became_active":"2024-07-27T14:42:33.224999+0000","last_became_peered":"2024-07-27T14:42:33.224999+0000","last_unstale":"2024-07-27T15:18:02.953153+0000","last_undegraded":"2024-07-27T15:18:02.953153+0000","last_fullsized":"2024-07-27T15:18:02.953153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:18:02.951586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:18:02.951586+0000","last_clean_scrub_stamp":"2024-07-27T15:18:02.951586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-29T02:17:10.501832+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":2824,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-07-27T15:17:58.066453+0000","last_change":"2024-07-27T15:17:58.067244+0000","last_active":"2024-07-27T15:17:58.066453+0000","last_peered":"2024-07-27T15:17:58.066453+0000","last_clean":"2024-07-27T15:17:58.066453+0000","last_became_active":"2024-07-27T14:42:32.853532+0000","last_became_peered":"2024-07-27T14:42:32.853532+0000","last_unstale":"2024-07-27T15:17:58.066453+0000","last_undegraded":"2024-07-27T15:17:58.066453+0000","last_fullsized":"2024-07-27T15:17:58.066453+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T15:17:58.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T15:17:58.064870+0000","last_clean_scrub_stamp":"2024-07-27T15:17:58.064870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T22:31:26.630429+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":18,"seq":77309411776,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76912,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294928,"statfs":{"total":96636764160,"available":96558006272,"internally_reserved":0,"allocated":5611520,"data_stored":2659637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":5611520,"data_stored":2659623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509882,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72756,"kb_used_data":5484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299084,"statfs":{"total":96636764160,"available":96562262016,"internally_reserved":0,"allocated":5615616,"data_stored":2659642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509886,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73140,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298700,"statfs":{"total":96636764160,"available":96561868800,"internally_reserved":0,"allocated":6008832,"data_stored":3057458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509888,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72748,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299092,"statfs":{"total":96636764160,"available":96562270208,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834575294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76908,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294932,"statfs":{"total":96636764160,"available":96558010368,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575288,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77300,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294540,"statfs":{"total":96636764160,"available":96557608960,"internally_reserved":0,"allocated":6008832,"data_stored":3057458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575291,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72748,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299092,"statfs":{"total":96636764160,"available":96562270208,"internally_reserved":0,"allocated":5607424,"data_stored":2659618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T15:18:22.046 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.148+0000 7ff2a40bb640 10 monclient: get_monmap_and_config 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.148+0000 7ff2a40bb640 10 monclient: build_initial_monmap 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.148+0000 7ff2a40bb640 10 monclient: monmap: 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T15:18:22.149557+0000 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T15:18:22.149557+0000 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a-legacy 2024-07-27T15:18:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.112:6789/0 mon.noname-c-legacy 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-b 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:3: v2:172.21.15.87:3300/0 mon.noname-a 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:4: v2:172.21.15.112:3300/0 mon.noname-c 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 1 Processor -- start 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 1 -- start start 2024-07-27T15:18:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 10 monclient: init 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.149+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff29c072ff0 addr v1:172.21.15.87:6789/0 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 1 --2- >> v2:172.21.15.87:3300/0 conn(0x7ff29c0b4650 0x7ff29c13ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c0b4650 addr v2:172.21.15.87:3300/0 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c0b2df0 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29c0ac610 con 0x7ff29c072ff0 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 1 -- --> v2:172.21.15.87:3300/0 -- mon_getmap magic: 0 -- 0x7ff29c1409b0 con 0x7ff29c0b4650 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff29c0ac300 con 0x7ff29c0b2df0 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.150+0000 7ff2a40bb640 10 monclient(hunting): authenticate will time out at 3442.176092s 2024-07-27T15:18:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.154+0000 7ff2a2658640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34716/0 (socket says 172.21.15.87:34716) 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 1 -- 172.21.15.87:0/1146151673 learned_addr learned my addr 172.21.15.87:0/1146151673 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 10 monclient(hunting): get_auth_request con 0x7ff29c0b2df0 auth_method 0 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.155+0000 7ff2a2658640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.156+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.156+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.156+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:22.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 10 monclient(hunting): handle_auth_done global_id 15590 payload 274 2024-07-27T15:18:22.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 1 -- 172.21.15.87:0/1146151673 >> v2:172.21.15.87:3300/0 conn(0x7ff29c0b4650 msgr2=0x7ff29c13ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:22.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 1 --2- 172.21.15.87:0/1146151673 >> v2:172.21.15.87:3300/0 conn(0x7ff29c0b4650 0x7ff29c13ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 1 -- 172.21.15.87:0/1146151673 >> v1:172.21.15.87:6789/0 conn(0x7ff29c072ff0 legacy=0x7ff29c0aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 10 monclient: _finish_hunting 0 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 1 monclient: found mon.noname-b 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.158+0000 7ff2a2658640 1 -- 172.21.15.87:0/1146151673 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c0aab70 con 0x7ff29c0b2df0 2024-07-27T15:18:22.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.161+0000 7ff2a2658640 1 --2- 172.21.15.87:0/1146151673 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7ff294009e50 tx=0x7ff29402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fe4ca6f0bbb5e2a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 1 -- 172.21.15.87:0/1146151673 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29402cb00 con 0x7ff29c0b2df0 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: dump: 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:22.174 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:22.175 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:22.175 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: _finish_auth 0 2024-07-27T15:18:22.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: _check_auth_tickets 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 1 -- 172.21.15.87:0/1146151673 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff29402cca0 con 0x7ff29c0b2df0 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a40bb640 5 monclient: authenticate success, global_id 15590 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a40bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:22.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a40bb640 10 monclient: get_monmap_and_config success 2024-07-27T15:18:22.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 1 -- 172.21.15.87:0/1146151673 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294034610 con 0x7ff29c0b2df0 2024-07-27T15:18:22.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a40bb640 4 set_mon_vals no callback set 2024-07-27T15:18:22.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:22.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:22.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.164+0000 7ff2a0e55640 10 monclient: dump: 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:22.178 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.179 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:22.179 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:22.179 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:22.179 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 10 monclient: shutdown 2024-07-27T15:18:22.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:22.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 -- 172.21.15.87:0/1146151673 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 msgr2=0x7ff29c140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/1146151673 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7ff294009e50 tx=0x7ff29402bf60 comp rx=0 tx=0).stop 2024-07-27T15:18:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 -- 172.21.15.87:0/1146151673 shutdown_connections 2024-07-27T15:18:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/1146151673 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b2df0 0x7ff29c140470 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/1146151673 >> v2:172.21.15.87:3300/0 conn(0x7ff29c0b4650 0x7ff29c13ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 -- 172.21.15.87:0/1146151673 >> 172.21.15.87:0/1146151673 conn(0x7ff29c0b8d70 msgr2=0x7ff29c0b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 -- 172.21.15.87:0/1146151673 shutdown_connections 2024-07-27T15:18:22.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.165+0000 7ff2a40bb640 1 -- 172.21.15.87:0/1146151673 wait complete. 2024-07-27T15:18:22.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 10 monclient: build_initial_monmap 2024-07-27T15:18:22.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 10 monclient: monmap: 2024-07-27T15:18:22.183 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.184 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.noname-a 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.noname-b 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.noname-c 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 librados: starting msgr at 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 librados: starting objecter 2024-07-27T15:18:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 Processor -- start 2024-07-27T15:18:22.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 -- start start 2024-07-27T15:18:22.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 librados: setting wanted keys 2024-07-27T15:18:22.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 1 librados: calling monclient init 2024-07-27T15:18:22.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.166+0000 7ff2a40bb640 10 monclient: init 2024-07-27T15:18:22.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient: _reopen_session rank -1 2024-07-27T15:18:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-27T15:18:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff29c072ff0 0x7ff29c0af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c072ff0 addr [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2024-07-27T15:18:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff29c0b2df0 0x7ff29c0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c0b2df0 addr [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): picked mon.noname-c con 0x7ff29c0b4650 addr [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c073550 con 0x7ff29c0b2df0 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c0733d0 con 0x7ff29c072ff0 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): start opening mon connection 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 -- 0x7ff29c14d630 con 0x7ff29c0b4650 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): _renew_subs 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.167+0000 7ff2a40bb640 10 monclient(hunting): authenticate will time out at 3442.193643s 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34730/0 (socket says 172.21.15.87:34730) 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 1 -- 172.21.15.87:0/3495908900 learned_addr learned my addr 172.21.15.87:0/3495908900 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 10 monclient(hunting): get_auth_request con 0x7ff29c0b4650 auth_method 0 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 10 monclient(hunting): _init_auth method 2 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.168+0000 7ff2a2658640 10 monclient(hunting): _init_auth creating new auth 2024-07-27T15:18:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.169+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-27T15:18:22.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.169+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-27T15:18:22.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.169+0000 7ff2a2658640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a1656640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff29c0b2df0 0x7ff29c0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 10 monclient(hunting): handle_auth_done global_id 15593 payload 824 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 -- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff29c072ff0 msgr2=0x7ff29c0af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff29c072ff0 0x7ff29c0af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 -- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff29c0b2df0 msgr2=0x7ff29c0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff29c0b2df0 0x7ff29c0ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 10 monclient: _finish_hunting 0 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 monclient: found mon.noname-c 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.171+0000 7ff2a2658640 1 -- 172.21.15.87:0/3495908900 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c0ae6f0 con 0x7ff29c0b4650 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.175+0000 7ff2a2658640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7ff294009e50 tx=0x7ff294004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.175+0000 7ff2927fc640 10 client.?.objecter ms_handle_connect 0x7ff29c0b4650 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.175+0000 7ff2927fc640 10 client.?.objecter resend_mon_ops 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294043020 con 0x7ff29c0b4650 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-27T15:18:22.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: dump: 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.196 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: _finish_auth 0 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: _check_auth_tickets 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff29402ce50 con 0x7ff29c0b4650 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 5 monclient: authenticate success, global_id 15593 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: handle_config config(0 keys) 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29403e690 con 0x7ff29c0b4650 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 10 monclient: _renew_subs 2024-07-27T15:18:22.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c0ae9d0 con 0x7ff29c0b4650 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff293fff640 4 set_mon_vals no callback set 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2927fc640 10 monclient: dump: 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:fsid a5c605d8-a191-47d5-9a8c-d25753c56cf8 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-07-27T14:39:24.184672+0000 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-07-27T15:18:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:1: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2: [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon.c 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 10 client.15593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 10 monclient: _renew_subs 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c0ad540 con 0x7ff29c0b4650 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.179+0000 7ff2a40bb640 1 librados: init done 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.180+0000 7ff2a40bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.180+0000 7ff2a40bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.180+0000 7ff2a40bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.180+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29c0aa460 con 0x7ff29c0b4650 2024-07-27T15:18:22.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.201+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff294046020 con 0x7ff29c0b4650 2024-07-27T15:18:22.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.201+0000 7ff2927fc640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 0x7ff28006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T15:18:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.202+0000 7ff2a1e57640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 0x7ff28006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T15:18:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.202+0000 7ff2a1e57640 10 monclient: get_auth_request con 0x7ff28006d810 auth_method 0 2024-07-27T15:18:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.202+0000 7ff2927fc640 10 client.15593.objecter ms_handle_connect 0x7ff28006d810 2024-07-27T15:18:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.202+0000 7ff2a1e57640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 0x7ff28006fcd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff298001560 tx=0x7ff298007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T15:18:22.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.205+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(1468..1468 src has 1140..1468) ==== 6357+0+0 (secure 0 0 0) 0x7ff294079d00 con 0x7ff29c0b4650 2024-07-27T15:18:22.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.205+0000 7ff2927fc640 10 client.15593.objecter ms_dispatch 0x7ff29c000b90 osd_map(1468..1468 src has 1140..1468) 2024-07-27T15:18:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.205+0000 7ff2927fc640 3 client.15593.objecter handle_osd_map got epochs [1468,1468] > 0 2024-07-27T15:18:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.205+0000 7ff2927fc640 3 client.15593.objecter handle_osd_map decoding full epoch 1468 2024-07-27T15:18:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.205+0000 7ff2927fc640 20 client.15593.objecter dump_active .. 0 homeless 2024-07-27T15:18:22.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.450+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186592 (secure 0 0 0) 0x7ff29404d030 con 0x7ff29c0b4650 2024-07-27T15:18:22.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.450+0000 7ff2927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-27T15:18:22.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.450+0000 7ff2927fc640 10 monclient: _finish_command 1 = system:0 2024-07-27T15:18:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.633+0000 7ff2a40bb640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-27T15:18:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.633+0000 7ff2a40bb640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-27T15:18:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.633+0000 7ff2a40bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.87:3301/0 2024-07-27T15:18:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.633+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7ff29c075a50 con 0x7ff29c0b4650 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff29407c360 con 0x7ff29c0b4650 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff2927fc640 10 monclient: handle_config config(1 keys) 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff293fff640 4 set_mon_vals no callback set 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff293fff640 10 set_mon_vals mon_health_to_clog = false 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff2927fc640 1 -- 172.21.15.87:0/3495908900 <== mon.2 v2:172.21.15.87:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7ff29407bc60 con 0x7ff29c0b4650 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff2927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-27T15:18:22.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.812+0000 7ff2927fc640 10 monclient: _finish_command 2 = system:0 2024-07-27T15:18:22.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 10 librados: watch_flush enter 2024-07-27T15:18:22.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 10 librados: watch_flush exit 2024-07-27T15:18:22.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 20 client.15593.objecter shutdown clearing up homeless session... 2024-07-27T15:18:22.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 10 client.15593.objecter successfully canceled tick 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 msgr2=0x7ff28006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 0x7ff28006fcd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff298001560 tx=0x7ff298007990 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 10 monclient: shutdown 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 msgr2=0x7ff29c0ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7ff294009e50 tx=0x7ff294004a40 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 shutdown_connections 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:6800/1547416426,v1:172.21.15.112:6801/1547416426] conn(0x7ff28006d810 0x7ff28006fcd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff29c0b4650 0x7ff29c0ace20 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff29c0b2df0 0x7ff29c0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 --2- 172.21.15.87:0/3495908900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff29c072ff0 0x7ff29c0af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 >> 172.21.15.87:0/3495908900 conn(0x7ff29c0b8d70 msgr2=0x7ff29c0afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 shutdown_connections 2024-07-27T15:18:22.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 -- 172.21.15.87:0/3495908900 wait complete. 2024-07-27T15:18:22.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-07-27T15:18:22.815+0000 7ff2a40bb640 1 librados: shutdown 2024-07-27T15:18:22.827 INFO:teuthology.misc:Shutting down mds daemons... 2024-07-27T15:18:22.827 INFO:teuthology.misc:Shutting down osd daemons... 2024-07-27T15:18:22.827 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-07-27T15:18:22.827 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:18:22.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T15:18:22.830+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35658) UID: 0 2024-07-27T15:18:22.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-07-27T15:18:22.832+0000 b485640 -1 osd.0 1468 *** Got signal Terminated *** 2024-07-27T15:18:25.056 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:27.922 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.058 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:27.924 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.073 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:27.939 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.105 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:27.957 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.106 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:27.958 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.108 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:27.974 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.109 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:27.975 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.120 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:27.972 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:25.124 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:27.990 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:28.930 INFO:tasks.ceph.osd.0:Stopped 2024-07-27T15:18:28.930 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-07-27T15:18:28.930 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:18:28.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T15:18:28.933+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35659) UID: 0 2024-07-27T15:18:28.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-07-27T15:18:28.934+0000 b485640 -1 osd.1 1470 *** Got signal Terminated *** 2024-07-27T15:18:30.128 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:32.994 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.197 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:33.063 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.230 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:33.095 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.250 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:33.116 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.251 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:33.102 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.251 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:33.103 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.255 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:33.121 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.262 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:33.114 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:30.273 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:33.139 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.143 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:48.008 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.145 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:48.011 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.160 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:48.026 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.198 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:48.050 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.202 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:48.053 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.204 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:48.070 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.210 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:48.076 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.219 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:48.070 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:45.226 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:48.092 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:47.240 INFO:tasks.ceph.osd.1:Stopped 2024-07-27T15:18:47.240 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-07-27T15:18:47.240 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:18:47.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T15:18:47.244+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35665) UID: 0 2024-07-27T15:18:47.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-07-27T15:18:47.246+0000 b485640 -1 osd.2 1474 *** Got signal Terminated *** 2024-07-27T15:18:50.141 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:53.006 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.145 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:53.011 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.170 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:38:53.036 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.189 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:53.040 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.211 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:53.062 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.211 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:53.077 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.214 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:53.080 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.221 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:38:53.073 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:50.225 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:38:53.092 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:18:53.342 INFO:tasks.ceph.osd.2:Stopped 2024-07-27T15:18:53.343 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-07-27T15:18:53.343 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:18:53.355 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T15:18:53.346+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35672) UID: 0 2024-07-27T15:18:53.356 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-07-27T15:18:53.347+0000 b485640 -1 osd.3 1476 *** Got signal Terminated *** 2024-07-27T15:18:59.445 INFO:tasks.ceph.osd.3:Stopped 2024-07-27T15:18:59.445 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-07-27T15:18:59.446 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:18:59.459 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T15:18:59.449+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35781) UID: 0 2024-07-27T15:18:59.462 INFO:tasks.ceph.osd.4.smithi112.stderr:2024-07-27T15:18:59.451+0000 b485640 -1 osd.4 1478 *** Got signal Terminated *** 2024-07-27T15:19:00.344 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:03.210 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.346 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:03.212 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.361 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:03.227 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.393 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:03.245 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.395 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:03.247 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.406 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:03.272 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.409 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:03.275 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.415 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:03.267 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:00.434 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:03.299 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:05.548 INFO:tasks.ceph.osd.4:Stopped 2024-07-27T15:19:05.548 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-07-27T15:19:05.549 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:05.564 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T15:19:05.551+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35785) UID: 0 2024-07-27T15:19:05.564 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-07-27T15:19:05.552+0000 b485640 -1 osd.5 1482 *** Got signal Terminated *** 2024-07-27T15:19:10.243 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:13.109 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.244 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:13.110 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.262 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:13.126 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.291 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:13.157 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.293 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:13.144 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.294 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:13.160 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.295 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:13.146 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.309 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:13.175 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:10.310 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:13.162 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:11.651 INFO:tasks.ceph.osd.5:Stopped 2024-07-27T15:19:11.652 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-07-27T15:19:11.652 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:11.665 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T15:19:11.655+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35787) UID: 0 2024-07-27T15:19:11.665 INFO:tasks.ceph.osd.6.smithi112.stderr:2024-07-27T15:19:11.656+0000 b485640 -1 osd.6 1484 *** Got signal Terminated *** 2024-07-27T15:19:17.754 INFO:tasks.ceph.osd.6:Stopped 2024-07-27T15:19:17.755 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-07-27T15:19:17.755 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:17.769 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T15:19:17.759+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35790) UID: 0 2024-07-27T15:19:17.769 INFO:tasks.ceph.osd.7.smithi112.stderr:2024-07-27T15:19:17.760+0000 b485640 -1 osd.7 1484 *** Got signal Terminated *** 2024-07-27T15:19:20.253 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:23.119 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.257 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:23.123 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.267 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:23.133 35360== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.303 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:23.154 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.303 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:23.155 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.304 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:23.170 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.305 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:23.171 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.317 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:23.168 35193== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:20.319 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:23.185 35366== Warning: unimplemented fcntl command: 1036 2024-07-27T15:19:23.857 INFO:tasks.ceph.osd.7:Stopped 2024-07-27T15:19:23.857 INFO:teuthology.misc:Shutting down mgr daemons... 2024-07-27T15:19:23.858 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-07-27T15:19:23.858 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:23.884 INFO:tasks.ceph.mgr.y:Stopped 2024-07-27T15:19:23.885 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-07-27T15:19:23.885 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:23.886 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T15:19:23.885+0000 7f7e0bd9f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35191) UID: 0 2024-07-27T15:19:23.886 INFO:tasks.ceph.mgr.x.smithi112.stderr:2024-07-27T15:19:23.885+0000 7f7e0bd9f640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-07-27T15:19:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-07-27T15:19:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-07-27T15:19:23.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-07-27T15:19:23.915 INFO:tasks.ceph.mgr.x:Stopped 2024-07-27T15:19:23.915 INFO:teuthology.misc:Shutting down mon daemons... 2024-07-27T15:19:23.915 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-07-27T15:19:23.916 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:23.941 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-07-27T15:19:23.929+0000 9fbc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35337) UID: 0 2024-07-27T15:19:23.941 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-07-27T15:19:23.931+0000 9fbc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-27T15:19:24.568 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:27.434 35360== 2024-07-27T15:19:24.568 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:39:27.434 35360== Exit program on first error (--exit-on-first-error=yes) 2024-07-27T15:19:24.592 INFO:tasks.ceph.mon.a.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-07-27T15:19:29.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-07-27T15:19:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-07-27T15:19:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-07-27T15:19:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-07-27T15:19:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-07-27T15:19:30.017 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-07-27T15:19:30.019 INFO:tasks.ceph.mon.a:Stopped 2024-07-27T15:19:30.019 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-07-27T15:19:30.020 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:30.035 INFO:tasks.ceph.mon.c.smithi087.stderr:2024-07-27T15:19:30.023+0000 9fbc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35339) UID: 0 2024-07-27T15:19:30.036 INFO:tasks.ceph.mon.c.smithi087.stderr:2024-07-27T15:19:30.025+0000 9fbc640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-07-27T15:19:30.629 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:33.494 35366== 2024-07-27T15:19:30.629 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:39:33.494 35366== Exit program on first error (--exit-on-first-error=yes) 2024-07-27T15:19:30.649 INFO:tasks.ceph.mon.c.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-07-27T15:19:34.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-07-27T15:19:34.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-07-27T15:19:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-07-27T15:19:36.122 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-07-27T15:19:36.122 INFO:tasks.ceph.mon.c:Stopped 2024-07-27T15:19:36.123 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-07-27T15:19:36.123 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T15:19:36.140 INFO:tasks.ceph.mon.b.smithi112.stderr:2024-07-27T15:19:36.126+0000 9fbc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35188) UID: 0 2024-07-27T15:19:36.141 INFO:tasks.ceph.mon.b.smithi112.stderr:2024-07-27T15:19:36.128+0000 9fbc640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-07-27T15:19:39.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-07-27T15:19:41.617 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:44.468 35193== 2024-07-27T15:19:41.617 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:00:39:44.468 35193== Exit program on first error (--exit-on-first-error=yes) 2024-07-27T15:19:41.636 INFO:tasks.ceph.mon.b.smithi112.stderr:daemon-helper: command failed with exit status 42 2024-07-27T15:19:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-07-27T15:19:42.225 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-07-27T15:19:42.226 INFO:tasks.ceph.mon.b:Stopped 2024-07-27T15:19:42.226 INFO:tasks.ceph:Checking cluster log for badness... 2024-07-27T15:19:42.232 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-07-27T15:19:42.351 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T15:19:42.351 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-07-27T15:19:42.493 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T15:19:42.493 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-07-27T15:19:42.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T15:19:42.557 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-07-27T15:19:42.648 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T15:19:42.648 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-07-27T15:19:42.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T15:19:42.698 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-07-27T15:19:42.794 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T15:19:42.795 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-07-27T15:19:42.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T15:19:42.981 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-07-27T15:19:43.056 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T15:19:43.056 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-07-27T15:19:43.124 INFO:tasks.ceph:Archiving mon data... 2024-07-27T15:19:43.125 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/data/mon.a.tgz 2024-07-27T15:19:43.126 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-07-27T15:19:43.144 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.NT4rfbLCGz 2024-07-27T15:19:43.145 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NT4rfbLCGz 2024-07-27T15:19:43.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.NT4rfbLCGz 2024-07-27T15:19:43.433 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.NT4rfbLCGz is 646KB 2024-07-27T15:19:43.537 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.NT4rfbLCGz 2024-07-27T15:19:43.552 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/data/mon.c.tgz 2024-07-27T15:19:43.552 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-07-27T15:19:43.606 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.856fgosTVB 2024-07-27T15:19:43.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.856fgosTVB 2024-07-27T15:19:43.790 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.856fgosTVB 2024-07-27T15:19:43.872 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.856fgosTVB is 646KB 2024-07-27T15:19:43.973 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.856fgosTVB 2024-07-27T15:19:43.988 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/data/mon.b.tgz 2024-07-27T15:19:43.988 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-07-27T15:19:44.007 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.l6nI39FSdO 2024-07-27T15:19:44.008 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.l6nI39FSdO 2024-07-27T15:19:44.216 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.l6nI39FSdO 2024-07-27T15:19:44.299 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.l6nI39FSdO is 646KB 2024-07-27T15:19:44.372 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.l6nI39FSdO 2024-07-27T15:19:44.387 INFO:tasks.ceph:Cleaning ceph cluster... 2024-07-27T15:19:44.387 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-27T15:19:44.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-27T15:19:44.747 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-07-27T15:19:45.016 INFO:tasks.ceph:Archiving crash dumps... 2024-07-27T15:19:45.018 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi087/crash 2024-07-27T15:19:45.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-27T15:19:45.044 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi112/crash 2024-07-27T15:19:45.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-27T15:19:45.069 INFO:tasks.ceph:Compressing logs... 2024-07-27T15:19:45.069 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-27T15:19:45.088 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-27T15:19:45.112 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-07-27T15:19:45.112 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-07-27T15:19:45.112 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42157.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108439.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42157.log: /var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39997.log 2024-07-27T15:19:45.113 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108439.log.gz 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44306.log 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39997.log.gz 2024-07-27T15:19:45.117 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40368.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41079.log 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39732.log 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42087.log: /var/log/ceph/ceph-client.admin.41079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35709.log 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109277.log 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.42157.log.gz 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39732.log: /var/log/ceph/ceph-client.admin.35709.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35709.log.gz 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39732.log.gz 2024-07-27T15:19:45.118 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42061.log 2024-07-27T15:19:45.119 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41870.log 2024-07-27T15:19:45.119 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42061.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-07-27T15:19:45.119 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109277.log.gz 2024-07-27T15:19:45.119 INFO:teuthology.orchestra.run.smithi087.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.44306.log.gz 2024-07-27T15:19:45.120 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41870.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-07-27T15:19:45.121 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.41079.log.gz 2024-07-27T15:19:45.121 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44443.log 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44443.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% 90.6% -- replaced with /var/log/ceph/ceph-client.admin.44443.log.gz 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41870.log.gz 2024-07-27T15:19:45.129 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36721.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40348.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41000.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.36721.log: /var/log/ceph/ceph-client.admin.41000.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36721.log.gz 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39512.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41614.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39512.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39512.log.gz 2024-07-27T15:19:45.130 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39736.log 2024-07-27T15:19:45.131 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-07-27T15:19:45.135 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-27T15:19:45.136 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35492.log 2024-07-27T15:19:45.139 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-07-27T15:19:45.139 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35652.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35492.log.gz 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35652.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35652.log.gz 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2024-07-27T15:19:45.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-07-27T15:19:45.141 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35409.log 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35572.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35409.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35409.log.gz 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35451.log 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-07-27T15:19:45.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35451.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35451.log.gz 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/valgrind/mon.b.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-07-27T15:19:45.149 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/valgrind/osd.7.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35612.log 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/valgrind/osd.5.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.35612.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35612.log.gz 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35353.log 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35353.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-27T15:19:45.150 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gz 2024-07-27T15:19:45.157 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-07-27T15:19:45.157 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41855.log 2024-07-27T15:19:45.157 INFO:teuthology.orchestra.run.smithi087.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39736.log.gz 2024-07-27T15:19:45.157 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-07-27T15:19:45.158 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41855.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-07-27T15:19:45.158 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39284.log 2024-07-27T15:19:45.158 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41172.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-27T15:19:45.158 INFO:teuthology.orchestra.run.smithi087.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gz 2024-07-27T15:19:45.158 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2024-07-27T15:19:45.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-07-27T15:19:45.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41820.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39284.log.gz 2024-07-27T15:19:45.159 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-07-27T15:19:45.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39211.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-07-27T15:19:45.159 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41494.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41855.log.gz 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39470.log 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41971.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2024-07-27T15:19:45.160 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2024-07-27T15:19:45.161 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2024-07-27T15:19:45.167 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44442.log 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39470.log: gzip -5 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44442.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39470.log.gz 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.41013.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40983.log 2024-07-27T15:19:45.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40163.log 2024-07-27T15:19:45.169 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40983.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.44442.log.gz 2024-07-27T15:19:45.174 INFO:teuthology.orchestra.run.smithi087.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-07-27T15:19:45.177 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-07-27T15:19:45.177 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2024-07-27T15:19:45.177 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7%/var/log/ceph/ceph-client.admin.42169.log: -- replaced with /var/log/ceph/ceph-client.admin.40163.log.gz 2024-07-27T15:19:45.180 INFO:teuthology.orchestra.run.smithi087.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-07-27T15:19:45.188 INFO:teuthology.orchestra.run.smithi087.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-07-27T15:19:45.189 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45094.log 2024-07-27T15:19:45.190 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-07-27T15:19:45.190 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45094.log: gzip 87.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2024-07-27T15:19:45.190 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41860.log 2024-07-27T15:19:45.191 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41930.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.45094.log.gz 2024-07-27T15:19:45.199 INFO:teuthology.orchestra.run.smithi087.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-07-27T15:19:45.199 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2024-07-27T15:19:45.201 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-07-27T15:19:45.201 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44305.log 2024-07-27T15:19:45.201 INFO:teuthology.orchestra.run.smithi087.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40983.log.gz 2024-07-27T15:19:45.201 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-07-27T15:19:45.202 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44305.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-07-27T15:19:45.202 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2024-07-27T15:19:45.203 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.41860.log.gz 2024-07-27T15:19:45.203 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35379.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-07-27T15:19:45.204 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-07-27T15:19:45.205 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41451.log: -- /var/log/ceph/ceph-client.admin.39998.log 2024-07-27T15:19:45.209 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41000.log.gz 2024-07-27T15:19:45.209 INFO:teuthology.orchestra.run.smithi087.stderr: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108691.log 2024-07-27T15:19:45.209 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44305.log.gz 2024-07-27T15:19:45.209 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39998.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2024-07-27T15:19:45.209 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1%/var/log/ceph/ceph-client.admin.108691.log: -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.y.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108691.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39728.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-07-27T15:19:45.210 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45092.log 2024-07-27T15:19:45.211 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2024-07-27T15:19:45.219 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44307.log 2024-07-27T15:19:45.224 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45092.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.45092.log.gz 2024-07-27T15:19:45.225 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-07-27T15:19:45.225 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-07-27T15:19:45.225 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41289.log 2024-07-27T15:19:45.225 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40646.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-07-27T15:19:45.228 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.44307.log.gz 2024-07-27T15:19:45.228 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-07-27T15:19:45.228 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41289.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-07-27T15:19:45.228 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.37723.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gz 2024-07-27T15:19:45.235 INFO:teuthology.orchestra.run.smithi087.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-07-27T15:19:45.236 INFO:teuthology.orchestra.run.smithi087.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41289.log.gz 2024-07-27T15:19:45.238 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39419.log 2024-07-27T15:19:45.238 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-07-27T15:19:45.238 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39419.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-07-27T15:19:45.239 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44944.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39419.log.gz 2024-07-27T15:19:45.241 INFO:teuthology.orchestra.run.smithi087.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/mon.a.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/valgrind/osd.3.log: -- /var/log/ceph/valgrind/mon.c.log 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/mon.c.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-07-27T15:19:45.256 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41212.log 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.2.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44794.log 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44794.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44794.log.gz 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40604.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-07-27T15:19:45.257 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-07-27T15:19:45.258 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41895.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz 2024-07-27T15:19:45.258 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-07-27T15:19:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41579.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-07-27T15:19:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109069.log 2024-07-27T15:19:45.282 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-07-27T15:19:45.282 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109069.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109069.log.gz 2024-07-27T15:19:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29511.log 2024-07-27T15:19:45.302 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39924.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44941.log 2024-07-27T15:19:45.302 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-07-27T15:19:45.303 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.29511.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29511.log.gz 2024-07-27T15:19:45.316 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-07-27T15:19:45.319 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44941.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44941.log.gz 2024-07-27T15:19:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108355.log 2024-07-27T15:19:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40438.log 2024-07-27T15:19:45.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108355.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108355.log.gz 2024-07-27T15:19:45.349 INFO:teuthology.orchestra.run.smithi087.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.41212.log.gz 2024-07-27T15:19:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-07-27T15:19:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40438.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40438.log.gz 2024-07-27T15:19:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108733.log 2024-07-27T15:19:45.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41086.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-07-27T15:19:45.360 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44444.log 2024-07-27T15:19:45.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108733.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108733.log.gz 2024-07-27T15:19:45.360 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-07-27T15:19:45.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44444.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.44444.log.gz 2024-07-27T15:19:45.364 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108397.log 2024-07-27T15:19:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41591.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-07-27T15:19:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108859.log 2024-07-27T15:19:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108397.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.108397.log.gz 2024-07-27T15:19:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44942.log 2024-07-27T15:19:45.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108859.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108859.log.gz 2024-07-27T15:19:45.371 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109153.log 2024-07-27T15:19:45.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44942.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.44942.log.gz 2024-07-27T15:19:45.375 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44233.log 2024-07-27T15:19:45.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109153.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109153.log.gz 2024-07-27T15:19:45.376 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-07-27T15:19:45.378 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44233.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44233.log.gz 2024-07-27T15:19:45.378 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-07-27T15:19:45.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39723.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-07-27T15:19:45.380 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-07-27T15:19:45.389 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-07-27T15:19:45.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39340.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-07-27T15:19:45.405 INFO:teuthology.orchestra.run.smithi087.stderr: 96.8% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-07-27T15:19:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35848.log 2024-07-27T15:19:45.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108649.log 2024-07-27T15:19:45.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35848.log: 88.1% 92.5% -- replaced with /var/log/ceph/ceph-client.admin.35848.log.gz 2024-07-27T15:19:45.429 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2024-07-27T15:19:45.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108649.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108649.log.gz 2024-07-27T15:19:45.432 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-07-27T15:19:45.446 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41979.log 2024-07-27T15:19:45.449 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44793.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2024-07-27T15:19:45.459 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-07-27T15:19:45.462 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2024-07-27T15:19:45.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41979.log: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.41979.log.gz 2024-07-27T15:19:45.469 INFO:teuthology.orchestra.run.smithi087.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-07-27T15:19:45.470 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108607.log 2024-07-27T15:19:45.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39733.log 2024-07-27T15:19:45.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108607.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108607.log.gz 2024-07-27T15:19:45.498 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42160.log 2024-07-27T15:19:45.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39733.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39733.log.gz 2024-07-27T15:19:45.500 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108523.log 2024-07-27T15:19:45.502 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42160.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42160.log.gz 2024-07-27T15:19:45.510 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-07-27T15:19:45.510 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108523.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108523.log.gz 2024-07-27T15:19:45.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-07-27T15:19:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41265.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-07-27T15:19:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-07-27T15:19:45.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-07-27T15:19:45.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41031.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-07-27T15:19:45.571 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109111.log 2024-07-27T15:19:45.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35682.log: 87.5% 92.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-07-27T15:19:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109111.log: -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 2024-07-27T15:19:45.585 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-07-27T15:19:45.590 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2024-07-27T15:19:45.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41261.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109111.log.gz 2024-07-27T15:19:45.599 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108565.log 2024-07-27T15:19:45.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41821.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108985.log 2024-07-27T15:19:45.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108565.log: -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 2024-07-27T15:19:45.617 INFO:teuthology.orchestra.run.smithi087.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-07-27T15:19:45.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2024-07-27T15:19:45.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108985.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108565.log.gz 2024-07-27T15:19:45.624 INFO:teuthology.orchestra.run.smithi087.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108985.log.gz 2024-07-27T15:19:45.642 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108901.log 2024-07-27T15:19:45.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39734.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2024-07-27T15:19:45.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108943.log 2024-07-27T15:19:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108901.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44232.log 2024-07-27T15:19:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108943.log: -- replaced with /var/log/ceph/ceph-client.admin.108901.log.gz 2024-07-27T15:19:45.665 INFO:teuthology.orchestra.run.smithi087.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108943.log.gz 2024-07-27T15:19:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-07-27T15:19:45.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44232.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-07-27T15:19:45.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40103.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-07-27T15:19:45.690 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44232.log.gz 2024-07-27T15:19:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108481.log 2024-07-27T15:19:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.44795.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-07-27T15:19:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-07-27T15:19:45.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108481.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108481.log.gz 2024-07-27T15:19:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40084.log 2024-07-27T15:19:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39731.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-07-27T15:19:45.739 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-07-27T15:19:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109027.log 2024-07-27T15:19:45.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41871.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40084.log.gz 2024-07-27T15:19:45.754 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-07-27T15:19:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2024-07-27T15:19:45.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109027.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109027.log.gz 2024-07-27T15:19:45.774 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-07-27T15:19:45.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40688.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2024-07-27T15:19:45.783 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108775.log 2024-07-27T15:19:45.788 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39735.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-07-27T15:19:45.791 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45093.log 2024-07-27T15:19:45.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.108775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-07-27T15:19:45.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.45093.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108775.log.gz 2024-07-27T15:19:45.810 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41745.log 2024-07-27T15:19:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40507.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.45093.log.gz 2024-07-27T15:19:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109237.log 2024-07-27T15:19:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41745.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-07-27T15:19:45.832 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41745.log.gz 2024-07-27T15:19:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109195.log 2024-07-27T15:19:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109237.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109237.log.gz 2024-07-27T15:19:45.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-07-27T15:19:45.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.109195.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109195.log.gz 2024-07-27T15:19:45.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-07-27T15:19:45.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39554.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-07-27T15:19:45.869 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2024-07-27T15:19:45.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39396.log 2024-07-27T15:19:45.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2024-07-27T15:19:45.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39396.log: 87.6% 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2024-07-27T15:19:45.892 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108817.log 2024-07-27T15:19:45.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41682.log: -- replaced with /var/log/ceph/ceph-client.admin.39396.log.gz 2024-07-27T15:19:45.910 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9%/var/log/ceph/ceph-client.admin.108817.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108817.log.gz 2024-07-27T15:19:45.913 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2024-07-27T15:19:45.993 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-07-27T15:19:46.057 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2024-07-27T15:19:46.490 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-07-27T15:19:47.457 INFO:teuthology.orchestra.run.smithi087.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-07-27T15:19:47.568 INFO:teuthology.orchestra.run.smithi112.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-07-27T15:19:48.448 INFO:teuthology.orchestra.run.smithi087.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-07-27T15:19:53.271 INFO:teuthology.orchestra.run.smithi112.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-07-27T15:19:53.363 INFO:teuthology.orchestra.run.smithi112.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-07-27T15:19:53.943 INFO:teuthology.orchestra.run.smithi112.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-07-27T15:19:54.418 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-07-27T15:19:54.531 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-07-27T15:19:54.724 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-07-27T15:19:54.824 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-07-27T15:19:54.827 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-07-27T15:19:54.827 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m9.726s 2024-07-27T15:19:54.827 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m44.259s 2024-07-27T15:19:54.827 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m1.555s 2024-07-27T15:19:54.945 INFO:teuthology.orchestra.run.smithi112.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-07-27T15:19:54.946 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-07-27T15:19:54.947 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m9.819s 2024-07-27T15:19:54.947 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m36.458s 2024-07-27T15:19:54.947 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m1.225s 2024-07-27T15:19:54.947 INFO:tasks.ceph:Archiving logs... 2024-07-27T15:19:54.947 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi087/log 2024-07-27T15:19:54.949 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-27T15:20:01.054 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi112/log 2024-07-27T15:20:01.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-27T15:20:06.385 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 1908, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-07-27T15:20:06.387 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-07-27T15:20:06.398 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 1908, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6e2a3aea75254d002e088a3ceee2a4fe2e292e3e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_57fd7103c78964675e1941b32d58041fb7c15af7/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-07-27T15:20:06.398 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-07-27T15:20:06.399 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-27T15:20:06.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-27T15:20:06.496 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-07-27T15:20:06.506 INFO:teuthology.task.clock:Checking final clock skew... 2024-07-27T15:20:06.506 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T15:20:06.509 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T15:20:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-07-27T15:20:06.526 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-07-27T15:20:06.527 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T15:20:06.528 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-07-27T15:20:06.528 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +959us[ +952us] +/- 59ms 2024-07-27T15:20:06.528 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 31 -1782us[-1791us] +/- 62ms 2024-07-27T15:20:06.528 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +768us[ +760us] +/- 66ms 2024-07-27T15:20:06.528 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +932us[ +917us] +/- 59ms 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -1819us[-1832us] +/- 62ms 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +733us[ +719us] +/- 66ms 2024-07-27T15:20:06.530 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T15:20:06.531 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-07-27T15:20:06.602 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-07-27T15:20:06.602 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-07-27T15:20:06.701 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-27T15:20:06.772 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-27T15:20:06.840 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T15:20:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T15:20:06.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T15:20:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T15:20:06.903 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-07-27T15:20:06.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-07-27T15:20:07.038 INFO:teuthology.task.internal:Duration was 3219.852452 seconds 2024-07-27T15:20:07.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-07-27T15:20:07.197 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-07-27T15:20:07.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-27T15:20:07.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-27T15:20:07.243 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T15:20:07.243 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T15:20:07.674 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-07-27T15:20:07.674 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-07-27T15:20:07.674 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-27T15:20:07.697 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-07-27T15:20:07.698 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-27T15:20:07.742 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-07-27T15:20:07.742 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-27T15:20:07.745 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-27T15:20:07.811 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-07-27T15:20:07.811 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-27T15:20:07.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-27T15:20:07.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-07-27T15:20:07.963 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-07-27T15:20:07.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-27T15:20:07.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-27T15:20:08.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-07-27T15:20:08.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-27T15:20:08.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-27T15:20:08.173 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-07-27T15:20:08.175 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-07-27T15:20:08.197 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-27T15:20:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T15:20:08.245 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-27T15:20:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T15:20:08.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-07-27T15:20:08.384 INFO:teuthology.task.internal:Transferring archived files... 2024-07-27T15:20:08.385 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi087 2024-07-27T15:20:08.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-27T15:20:08.442 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-07-21_21:00:02-rados-squid-distro-default-smithi/7811677/remote/smithi112 2024-07-27T15:20:08.442 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-27T15:20:08.476 INFO:teuthology.task.internal:Removing archive directory... 2024-07-27T15:20:08.476 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-27T15:20:08.478 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-27T15:20:08.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-07-27T15:20:08.550 INFO:teuthology.task.internal:Not uploading archives. 2024-07-27T15:20:08.550 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-07-27T15:20:08.794 INFO:teuthology.task.internal:Tidying up after the test... 2024-07-27T15:20:08.794 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-27T15:20:08.796 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-27T15:20:08.810 INFO:teuthology.orchestra.run.smithi087.stdout: 266040 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 27 15:20 /home/ubuntu/cephtest 2024-07-27T15:20:08.815 INFO:teuthology.orchestra.run.smithi112.stdout: 266040 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 27 15:20 /home/ubuntu/cephtest 2024-07-27T15:20:08.815 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-07-27T15:20:09.010 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-07-27T15:20:09.157 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-07-27T15:20:09.159 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3219.8524520397186 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-07-27T15:20:09.160 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-27T15:20:09.238 INFO:teuthology.run:FAIL